2D to 3D Conversions

2D to 3D conversions

2D to 3D conversion is the process of converted a film from 2D (normal film) to a 3D (or stereo) film for viewing with a stereographic viewing system. (polarized glasses, shuttered glasses, special screen, etc)

Conversion had gained a bad name after Clash of the Titans and a few other films that had been rushed through at the last minute. Just as you can’t do a 2000 shot vfx show in 6 weeks with great quality, you can’t do a quality conversion of an entire movie in 6 weeks.

Most people think of a 2D to 3D conversion as a factory type of operation but it’s far from it if it’s to be done right.  Much of my past year was spent working for Legend3D dealing with some interesting and very technically challenging problems. Legend had completed work on Alice (Tea party sequence and Drink Me sequence) and done the conversion on the first 3 Shrek films to be released for 3D TV.

Legend was starting on Transformers: Dark of the Moon and would need help with the extensive visual effects shots. Legend 3D ended up converting over 77 minutes of footage for Transformers.  Most of the vfx shots were converted in a 4-5 month time span. The converted footage was a mix of visual effects shots (from ILM and DD) and non-vfx shots.  In most cases these would be shots intercut back and forth with original stereo footage shot on location.  This raised the complexity much more than if the entire film had been converted. Just as doing vfx in a live action film is a different task than creating an animated film (or virtual sequence).

This was also more difficult than doing a library title or a film that was completed.  The same issues vfx companies face of ever changing edits and creative directions was in full force. In addition the vfx shots were changing and so we had to follow very closely and track all changes such that we could turn around the final converted shots within a day or two of deliver of the last vfx shots.

While at Legend I helped them to develop the pipelines required to do this type of work efficiently and at as high of quality level as possible. Legend had some proprietary software and techniques for different stages in the process. I wrote a number of specialty Nuke plugins and scripts to help leverage these proprietary software packages with existing software. I worked with a number of artists and developers at Legend to review and analyze the shots as they were delivered and worked closely with ILM and DD to get the proper elements that would be required. Both technical and creative issues had to be examined to get the best possible quality. Overall this represented a number of challenges that were unique to the world of stereo conversion and we were able to raise the quality of conversion to a new level.

In what follows I’ll cover the basics of the conversion process. I won’t be able to go into specific detail to due the proprietary nature but hopefully this will provide an understanding of the process.

Conversion versus shooting stereo
Many people are under the mistaken belief that shooting stereo is superior to conversion in every way.  The key to getting great 3D is to design, shoot and edit for 3D, even if it will be converted. If you shoot stereo but don’t consider the 3D world then you will have problems.

In its current form 3D is a stereo process, which means creating imagery for each eye. It’s actually a big cheat when projecting 2 different images to simulate the look of the 3D because your eyes are always focused on the screen but objects appear at other depths. This is a fairly unnatural process.

The following are some of the basic pros and cons. As with everything there are tradeoffs to be made. Shooting stereo does capture the images in 3D but many of the parameters are locked in at the time of shooting. Shooting stereo also requires a fair bit of post work to actually get the images to match. Films that are shot in stereo will still require a certain number of conversion shots just due to problematic shots or limitations while shooting. This was the case even with Avatar.

Shooting Stereo – PROS
• Capturing stereo images at time of shooting
• Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.
• Preview stereo images on set and on location
• Footage can be edited and reviewed in stereo context with the correct equipment

Shooting Stereo - CONS
• Requires specialized camera rigs
• Camera rigs contain 2 cameras so are larger than non-stereo camera setups
• More time required to shoot (higher shooting costs)
• Requires shooting on digital cameras (no option of film)
• Special stereo monitor and glasses required on set to check settings.
• Requires a certain amount of adjustment, alignment and cleaning.
• Requires special rig technician to handle these adjustments.
• Restrictions on lenses that can obtain good looking stereo
• Requires locking in stereo depth at time of shooting. This cannot be adjusted in post.
• Shooting with ‘toe in’ (non-parallel) camera systems requires a convergence puller similar to a focus puller.  Having the correct convergence point is critical when editing shot to shot.  This convergence point can only be adjusted so far in post. (With parallel camera systems this convergence is all set in post.)
• Lens flares and shiny reflections can appear different to both cameras and will require post work to correct to prevent viewing problems of final film.
• Because of the polarizing through the beam splitter, water and even asphalt with a sheen will appear different to both cameras and require post correction
• Post work required to align and fix footage. Since one camera is shooting through a beam splitter and another is shooting the beam splitter reflection there is typically an imbalance in terms of color, contrast and alignment.
• Visual Effects require more work and more rendering (added time and cost)

Stereo Conversion - PROS
• Ability to shoot with standard process (Composing with 3D in mind still recommended)
• Choice to shoot on film with any film camera or on digital with any digital camera
• Unlimited choice of camera lenses (Extreme telephoto still not recommended for best 3D results)
• Ability to add depth and volume even on telephoto shots
• Ability to set the 3D depth in post on a shot by shot basis
• Ability to set the convergence as desired for the edit
• Flexibility to adjust the depth and volume of each actor or object in a scene as a creative option not available when shooting stereo.
• Visual Effects are handled in the standard way

Stereo Conversion - CONS
• Extra time required after edit to properly convert or sequences need to be locked during edit to allow conversion to take place while in post.
• Added cost of process
• Reflections, smoke, sparks and rain more difficult to convert but not impossible.



The 2D to 3D process
Each company handles conversion in a slightly different way, sometimes with proprietary tools and techniques but what follows are some of the fundamentals.

The first step in the conversion is to have a stereographer review the shot and determine how it will work in 3D space. How much depth budget there will be (how much total depth in the scene) and where the convergence point will be.  The convergence point ends up on the movie screen and imagery in front of the screen is said to be in negative parallax and imagery behind the screen is positive parallax.  If you have two images (left and right eye) of a circle and it’s in the exact same spot for both eyes then it will appear on the screen surface. If you offset the images of the circle toward or away (left/right) from each other so they overlap but don’t match then you’ll have the circle float in front of the screen or behind the screen depending on the direction of offset. The amount of offset will determine how far forward or back it appears.

Convergence decisions are based on the action and edit as well as what is in the scene. Frequently the key actors eyes are used for the convergence point so when cutting from one scene to another the audience doesn’t have to visually look back and forth constantly.

With this information the image is broken down. How much detail depends on where it is in space.  Stereo eyesight primarily works within 20-25 feet. As objects get further away there tends to be less sense of 3D stereo. Breaking down the image may require roto or some form of image area selection. If you’ve done roto or frame by frame extraction you know how difficult and time consuming it is. But in this case everything in the images is broken down, not just a foreground actor. In vfx if you roto a group of people you may be primarily concerned about the group areas that overlaps the area to be replaced. With 2D to 3D you have to isolate each person and each object on a different depth plane.

So imagine the degree of difficulty extracting every 3D surface from a 2D image and making all the frames and transitions match perfectly and smoothly including wild camera moves and explosions with thousands of sparks. Depending on the exact 2D to 3D process used you may have build actual 3D models or at least replicate that process to some extent. If there is a table in the image the table-top will have to be split into top, front and side.

The amount of work and complexity means that if there is a change to the image (edit, reposition, speed up, slow down, etc) it may require redoing the entire shot.

With traditional roto you can get away with things like losing a few fly away hairs on an actress. If they’re clipped off and a new background is composited no one will know. With conversion however any hairs that have not been extracted may end up at the wrong 3D depth. Imagine a character in the foreground with long flowing hair but having the hair stuck to the mountains in the background. Each company leverages what techniques and software they can to help with this process and compositors can also help the process by leveraging various extraction methods.

3D settings
Once the image is actually broken up it is necessary to create the depth. A conversion stereographer may use propriety software or heavily modified Nuke or other application to help with this step. This can be in a true 3D world or it can be simulated 3D with gap offsets.  The stereographer usually works with a stereo monitor and adjusts and animates the very shapes, objects and planes to move in 3D space. This needs to match the surrounding shots and the 3D relationships need to be correct. In the case of Transformers the volume and depth of the objects has to match to the original stereo photography. The show stereographer works with the director as well to provide both creative and technical feedback regarding the depth range and convergence planes.  Cory Turner was the show Stereographer on Transformers.

Viewing stereo footage and making adjustments requires developing an eye for stereo. Not everyone can do it and different people have different aptitude for judging stereo. With most vfx work it’s usually obvious when there are matte edges or a color balance problem between elements. Seeing these types of problems is consistent across a number of individuals. Yet viewing stereo footage is much more subjective. Even stereo experts commonly disagree on the latest 3D film. Some may say it’s the best stereo they’ve ever seen and others will say it’s the worst they’ve seen. Technical issues and creative choices by the stereographer and director will affect the final results.

But even at this stage if all you’ve done is cutout the objects you’ll end up with flat images set at different depths. What you need to do is add volume and real 3D shape to the image that matches. This is one of the areas that 2D to 3D conversion provides an advantage over shooting stereo. Telephoto lens images can be provided more volume than would be possible shooting stereo. Different objects can be given more or less 3D emphasis. An actor needs to have their head rounded, their eyes sunken in and their nose coming out.  This relationship needs to match even as the actor turns his head and the camera moves. There’s a real art to the process and much of this can not be automated. The skill of the artists combined with the tools at their disposal has a lot to do with the final results.

The end result of this process is 2 images – a left and right eye image. If this is a full 3D process there are essentially two 3D CG cameras setup with an InterAxial offset (lens separation just as if shot with a stereo camera system). Some companies use the original image as one eye and generate only the other eye. Legend tends to convert the original footage as if it’s the center image and so they actually generate both eyes. This tends to make sure both eyes match and the image itself isn’t always offset from the original framing.

Because the original image was 2D when sub-images are offset they create gaps where there is no image. The simple way of thinking of this is to start with 2 identical images and cutting out an actor in each. Offset one cutout of the actor 10 pixels to the right and offset the other 10 pixels to the left. Now there’s no image on the edge where the image was moved from. This gap needs to be filled in so you don’t see a black edge in the finished shots. If the gap is really small you can cheat it but these artifacts can be quite noticeable on the big screen. To do it right in most cases requires treating the edge of each object as a full on rig removal. Creating or shooting a clean plate can greatly help with this process.

Keep in mind all of these steps, adjustments and cleanup have to be done on every single shot in the film if the film was originally shot 2D.

Reflections, lens flares, smoke and other problems
And as if that wasn’t tough enough imagine the main actor wearing glasses. The glasses have to be set at the correct depth to relate to the head. Any thumbprints or dirt on the lenses have to be on that same plane, however the actor’s eyes have to be pushed in. More than likely there is a reflection on the glasses of the room, which may be 20 feet away. And the actor may have a highlight on his eye that reflects something 5 feet away.  Each of these different images have to be extracted and removed so you end up with a clean eye with no reflections, the glass reflections, the eye highlight and the thumbprint. This is from live action so requires the compositor to go through a process along the lines of rig removal but splitting in to each depth layer. Each of these layers then has to be placed and positioned in 3D and then composited back together.  If this isn’t done then it will look like the actors eyes and the reflections are simply painted on the surface of the glasses. Needless to say this looks is very wrong in 3D.

Same issues if the shot has glass window, mirror, chrome or other reflective surfaces. The issue of seeing partially transparent layers at different depths also comes into play if you have smoke or fog in the scene. If you have partially transparent smoke in front of a building you don’t want the building to be pulled forward to 10 feet from camera nor do you want the smoke to be pushed all the back to the front of the building.  Same exact problem happens with lens flares.  Each of these has to be removed into a separate element and then added back in at the right depth, usually near the screen plane for lens flares.

Transformers of course was full of fast action with glass windows, smoke, explosions, sparks, and lens flares.

Conversion of VFX shots
VFX shots are usually tough enough without having to deal with two slightly different views for the left and right eye. Working in stereo takes more time and adds an additional level of complexity. When the footage is shot as 2D then the vfx can be finished as 2D shots and approved by the director, which tends to be quicker and easier. But it’s usually best to take advantage of the vfx process when doing the conversion to produce better conversion quicker.

In vfx shots it’s not unusual to add smoke and lens flares among other things. Obviously once these are baked into the shot the conversion artist will have to re-separate them. It makes more sense for speed and quality to simply breakdown the composites in such a way to keep key layers separate and use those layers to set the depth in the conversion stage. Some of the other vfx elements are also useful such as clean plates and potentially some of the roto or mattes.

For shots with CG elements it’s better if the CG elements are rendered separately so there’s no need to extract them and fill in the gaps if only a finished composite was provided. Taking this further the vfx company can render depth maps so the conversion company can leverage it and provide a consistency to the creature (or in this case, transformers)

The majority of 2D vfx shots in Transformers was done this way.  The sheer complexity of the Transformers, the number of moving and changing surfaces, really took advantage of this process and allowed them to be consistent with the stereo vfx shots. A few shots were done where the vfx company rendered 2 views of the transformer and then Legend created the 3D world to go around the rendered stereo image. In a few other cases Legend converted the background to stereo and the vfx company worked from that.

Tips for VFX companies
If the film project you’ll be starting is to be converted make sure to work with production and the conversion company. The conversion company should provide a list of requested elements and the ideal formats. This isn’t simply a matter of turning over a Nuke script.  Chances are your company has proprietary plugins and scripts. It’s also likely that the composites may be very complex yet the conversion company only needs the layers that make sense in depth. You may not typically render depth maps and potentially other elements independently.  The cost and time to do this re-purposing of composites and elements will have to be accounted for either in the vfx bid or as a separate bid. If it’s vfx heavy show it’s likely there will be a small team of people prepping images for the conversion process.

Ideally the production company hires a show stereographer who oversees the process from shot design through to completion.

As with the original photography it’s important for everyone to consider the work as if it were in true 3D space.  No more cheating where animated characters are placed or using one element that is supposed to represent multiple depths. These problems will show themselves once converted to real 3D space.

Compositors need to consider building the composite based on 3D space. When holdouts are built in or there are elements with an odd layer order then it becomes difficult to re-build the composites to work in a dimension conversion.

Work out a naming process for these special conversion elements to make it easy for the conversion company to rebuild the composites.   Example: RL045.layer_1.over.00128.exr

It’s also important to provide any special lookup tables or color specific nodes

Summary
Conversion can create successful stereo images that match images shot in stereo as long as experienced artists are allowed sufficient time and control to do this complex and vfx heavy process correctly. Just as with vfx companies, the selected conversion companies will determine how successful the final results are. Neither visual effects nor conversion are truly commodities that can be randomly assigned to other companies with the same result.

Legend
Legend3D also did at least some of the conversion work on the following features while Transformers 3 was being done: Green Hornet, Pirates of the Caribbean: On Stranger Tides, Green Lantern, Priest,  Smurfs3D.  They were able to leverage many of the improvements created for Transformers 3.

[Update: Forgot to post a link to a VFX Guide Podcast that touches on the conversion.  I believe it's touched on at about 45+ minutes in.]

[Update: I was reminded when responding to a comment below there was a video of Michael Bay and James Cameron discussing shooting stereo.  This was done after the film had been shot and within a few weeks of release.  Video of stereo conversation of Bay and Cameron]

[Update: I got a few tweets from @philwittmer that I thought I'd address here since there's still seems to be some confusion.

Phil:

Sorry but that all seems biased towards 2d conversion. With no mention of roto time. granted its an interesting read I never thought of it like that but.... a couple of major points here need to be made
1. U get limitless depth if shooting 3d as opposed to designated planes
2. Moving foc point in post just means bad directing
lastly and most importantly. The conversion will only be as good as the budget and time. Which unfortunately is usually bad

Me:
Yes, any conversion takes time. Roto takes time, extractions take time, setting depth takes time.
That's why I stated you can't do it in 6 weeks and that was listed as one of the CONs of doing conversion. It is a time consuming process.

If the film has a large number of vfx shots then this conversion can be done (especially on the non-vfx shots) while the vfx are being worked on.  If the vfx shots are locked and can be worked on as you proceed it still takes time but that post time is already used up by the vfx work. That means it doesn't necessarily mean a large delay after the vfx, depending on the show.

Keep in mind all stereo shoot footage needs to be processed as well. Every shot will have to go through a process to be adjusted for color and alignment and other corrections so the left and right images match correctly. This is not an overnight process and so you need to budget time to do shot by shot corrections of the entire film, like DI but can be much more intense. If any shots are truly broken they may need 2D to 3D conversion or serious vfx work to repair them so expect even more time.  This will be less than doing a conversion of the whole film but it's time that should not be discounted.

Unlimited 3D - When shooting 3D you have to determine the amount of depth you want.  And that is not unlimited. You can't have something 1 foot from the viewer and things pushed to infinity without compressing that space.  If you have too much separation in the background you'll get divergence where people's eyes will go walleye (each looking outward). Not pleasant. Too much negative parallax will cause eye strain as well.

Now you can get nice in camera stereo of complex things in the mid distance such as tall grass, leaves on a tree, smoke, etc.  That's why that's a PRO for shooting stereo:  Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.

Designated planes - Just to be clear this isn't like a multiplane camera when you have flat cards at different distances and there are 10 designated planes.  It's not like the process that seemed to result in some viewmaster displays (i.e. all images flat to camera at different depths). Any flat surface can be at any angle. It has to be to actually work in a 3D movie. That road has to be angled away from camera, that building on the side going away from camera, etc. And more importantly you're creating a dimensional space, at least with the better processes.  You have to create the sense of roundness and volume as if you built it as a real 3D model. A shot of a head is a good example (a complex sphere with nose, eye sockets, ears, jaw, etc). You can not get away with having a person on a flat card and expect that to hold up for the audience. Whether the company uses real 3D models, depth maps, complex meshes, etc. you can be sure it's not made up of just designated planes.  And yes, that make it very hard, especially for the items that I listed as easier to shoot. Numerous, complex shapes that are changing in 3D space are very difficult.

During shooting of stereo you have the issue of setting the IA (interaxial)  to set the overall depth. And you have the issue of setting the convergence.  Either of these may be manually changed during the shot depending on the end result.  Is it bad for a director to find out later a different edit works better? What if in planning the shots the director originally thought the shots would cut together in a specific way and shot with that in mind. But once the cuts are changed the conversion may have to be adjusted in post so there won't be as much eye strain. Wide shot to closeup, the timing of the convergence wasn't right, etc. What if the initial IA was high but the fast action caused more strain? The director and stereographer have to really do some planning and testing before shooting. And even then there may be an edited version that doesn't work and needs to be adjusted. And some things like IA can't be adjusted cleanly in post currently. So some 3D decisions made on set are what will be.

>The conversion will only be as good as the budget and time.
This goes for every element of a film. You can't do Avatar in 10 days for $10. You can't record sound for a feature film with your iPhone. If the studio doesn't allow the correct amount of time and money to do the conversion correctly then they will have a movie they spent a lot of money on looking terrible in 3D. And this will cause turning more people away from 3D. The studios are starting to understand the correlation between good 3D and a happy audience willing to pay for 3D movies. And many of these 3D films (shot or converted) are large tent pole movies so there is already a large budget so the studios don't want to gamble and go super cheap.

In the end there are pros and cons to both methods. One of the main points of this article is both can produce good results. Which one to use and what balance of the two depends on the specific project. If production and the director can commit to shooting 3D with a real stereographer and take the necessary care and attention then shooting stereo may be their best choice. Just don't rule out 2D to 3D conversion as one of the options to consider simply because it's been done poorly on some films.

Update: Opensource 2D/3D conversion Gimpel3D for those interested.