Sunday, September 16, 2007

Naming conventions and workflow

Naming conventions and workflow.

Since this question has come up a few times I’ll try to address it here. Note that if a company is already up and running chances are they have a workflow in place along with naming conventions. I’ll describe some of the more typical approaches but there are no standards in the industry.

It’s important to try to standardize naming conventions and also directory configurations since you’ll have dozens or even hundreds of files associated with each shot (All the live action pieces such as BlueScreen, background, dust elements, explosions, each model will have multiple files and associated texture maps, each shot will have multiple animation files, shaders, composting scripts, etc) Multiply those by the number of takes or versions and then multiply that by the number of shots in film (hundreds or thousands). That’s a lot of data to manage.

When the initial movie script arrives it usually hasn’t been broken down by production yet so that means when bidding you have to assign shots your own names. Even if the script is broken down it’s only broken down to the scene level. Each scene may have a large number of shots. From a live action standpoint these shot numbers are assigned during shooting and usually relate to the shooting order (i.e. 17A over shoulder, 17B extreme closeup, etc) . This allow the live action side of the project to be organized (script supervisor notes, asst camera slates and camera reports, editorial, etc) But by the time the shooting starts the VFX team has already created storyboards and possibly previs with numbers along with a budget breakdown of each and every shot. These days you may also have a number of pure virtual shots (no live action) such as all CG or pure matte painting shots.

Typical method in VFX is to assign a 2 or 3 letters ID to each sequence that at least might help people to know the sequence. If you had a rocket landing on the moon sequence you might label it RLM (Rocket Lands Moon) (Try to label by the gist of the scene rather than the location since the location may change and you’ll forever have to explain to people what the ID used to mean and how it relates now) From there as you create the storyboards you increment the shot numbers by 10 to allow adding new shots in-between as the director and artist modify the sequences. These are the numbers that are used in the bids and schedules and will hopefully remain through the production.

When shooting the VFX people work with the script supervisor. Somewhat standard is to place a V in front of the slated shot number to note that it’s a visual effects shot. If space on the slate permits the asst cameraman may note the VFX number on the slate as well. The script supervisor and the VFX coordinator keep a running cross reference of the VFX shots names and the live action shot IDs. Note that in editing a shot may be re-used or used for an entirely different shot.

Once a sequence is edited and locked the editorial department provides information on the shots. They used 83B, take 5 for this shot and it starts and ends at specific frames. (Based on keycode on the film or based on the Avid timecode info) If it’s a large show the VFX companies may have their own editors who get a copy of the avid editing bins which they breakdown. The negative (assuming it’s shot on film) will be shipped from the lab to whoever is scanning the film. The VFX editor works with the scanning company regarding the label process. In some cases the VFX company may scan it themselves but now a lot of this work is done by scanning companies.

The digital files are delivered to VFX company by high-speed connections or hard drives or other techniques. Each VFX may re-name them for their internal naming conventions when the files are brought online. Because of the massive amount of data not all shots and elements are ‘online’ at the VFX house. They may be on high-density tape storage or other storage system and brought on by support as needed. There’s usually a whole team of people doing this.

A shot may have the name of rlm0030.cc.dr
The cc might stand for color corrected. Dr might be for dirt removal.
bs might be for bluescreen or they may choose to label the shots a,b,c, etc for each live action element. In addition to standard suffixes for the file themselves (.exr, .cin, .jpg)
each frame will have a number. Numbers may be for a fixed number (4 or 5) of digits (i.e. myshot.0001.jpg, myshot.0002.jpg) or may float (i.e. myshot.1.jpg,myshot.2.jog) Even the idea of leading zero or non-leading zero will make a difference depending on your software.

Most VFX work is done on a frame level rather than passing QuickTimes around. This allows work on frames such as compositing even while other frames are being rendered. It also allows spreading the rendering/compositing over multiple machines and more flexibility in file formats.

As the shot goes through the different stages the complexity grows. Matchmoving (or layout) may need a couple of passes to get their basic animation information correct. That’s take 2 of that file. The animator might be working on take 5 but the Technical Director is rendering take 4 (from the day before) The model may have different version numbers since production may start before the CG models are finalized (or someone (the director, may change it)). It’s possible on take 8 of the final render the director decides he actually liked the animation of take 3 better but with the new lighting and render of take 8. It’s also likely you’ll have multiple people messing with files at the same time (animator, compositor, TD)

This soon turns into a nightmare if you’re not on top of it. The typical approach is to standardize on a directory structure. Details tend to be unique to each company but you may have a shot folder which will hold an animation folder, model alias or reference folder, composite folder, etc. Each of these will have sub-folders of different types of files or work. This directory is then configured on any machines where you may want to spread the work. You may want to render and composite frames 1-100 on computer 1 and 101 -200 on machine 2. Or it may be on a special render farm system. In many cases you’ll be using aliases or pointers to the actual image or data files so they can be in one place. That minimizes having to initially copy over all large files but it does mean they may have to be moved during the render process/composite.

So to deal with all of this type of data each VFX company has written multiple databases and scripts. These may be UNIX script, Python, or a number of scripting languages. Databases can be FileMaker on up full-blown custom code. There are now some off the shelf type of products (or modified off the shelf products). Luckily I personally don’t tend to have to get into the nitty gritty. Some of these problems are similar to programming large projects where you have version control to allow people to access files yet still allow it to lock out people or to merge the differences.

During the course of the day each person works on his specific area and then they submit a request. The TD may run a test frame and calculate the number of process hours required to do the shot. All of these requests are submitted and the CG supervisor may review this list with the producer and the supervisor. Since there may not be enough processors and time (likely the case) they will have to decide what the priorities are supposed to be. Some shots may be on a slow render to be finished in a few days and other shots may need to be done the next day for the director to review. Some shots may opt to be done as plastic renders or without fur just for checking.

The script or software will then distribute the shots across multiple machines and be sure to grab the latest version of the animation and model, etc.

In some cases editorial has scripts to automatically assemble all the new renders into the cut. Additional software may be used to keep track of daily notes from the supervisor and the director.

Individuals or new companies starting out should review their software packages and see what they can import and export in regard to frame numbering. Also review your software to see if and how it handles multiple computer networking. (Whether it’s After Effects or a full rendering package). Keep an eye on the number of characters you can have in a filename. Create scripts and database programs that can deal with this naming and directory structure. Test it out on small test projects.

[Update 12/14/2012
Steve Molin provided a directory recommendation he uses.  Steve was a key CG person at ILM, LAIKA, and Image Movers]

from Steve:


Over the years that computer have been used in creating graphics for film and video, the question has come up time and again: how do we lay out the directories on the hard drive so that we can all work together efficiently. What follows is the way I like to see it done, based on my years in the industry - Steve Molin
$SHOW
assets
$TYPE
$NAME
modeling
rigging
surfacing
mattePainting
shaderDevelopment
sequences
tools
shaders
common (a sequence with data common to all sequences)
$SEQ
$SHOT
common (a shot with data common to shots in this seq)
capture
animation
characterFinaling
lighting
compositing
mattePainting
dynamics (other than charFin, eg dirt, splashes, flames)
rotoscoping
  1. If a shot ends up with eg two lighters working on it, the second and subsequent would get appended the username, eg light_smolin.
  2. it is preferable to create directories only when needed to reduce number of empty dirs
  3. sequences to be named with mnemonic codes, eg rr for raptor rotunda
  4. shots to be named with sequence and number, eg rr1, rr2 etc. “Count by ten” allowed, eg  rr10, rr20 to facilitate insertions if desired.
  5. development is to occur in developer-specific shotdirs, eg sequences/rr/dev_smolin
  1. Each of the lowest level would have the same structure, as below:
maya (under here, maya has complete control)
images
source (ie input from another discipline)
reference
renders (ie output from this discipline)
scripts (eg Nuke scripts, Katana scripts … Maya scenes?)
tools (eg Python scripts, compiled binaries)
shaders
curves (eg animation curves)
points (eg point clouds, brick maps)
assets (copies from assets branch, to make shotdirs self-contained)
models
rigs
surfaces
digitmattes


Related links:
RaysInBlue Blog has a great set of links on VFX and Animation pipelines
Art of CG Supervision goes into extensive details

5 comments:

  1. Hi, Scott:

    Thanks you for deal with this issue, I see there is a lot of experience behind this...

    I would like to ask you... can you let me translate your blog to spanish ?

    If that is the case, can I publish that translation on my blog (referencing to yours, of course)?

    In any case, thanks a lot for your efforts

    ReplyDelete
  2. Sure, you may post it to your site as long as there's the copyright and linkback. Thanks.

    ReplyDelete
  3. Thanks to you and your great work with this blog. Your insights about the visual effects industry are very valuable.

    I'll be posting your articles soon at pelisdegaraje.blogspot.com .

    The very act of translating make me much more aware about all that stuff.

    ReplyDelete
  4. Hi Scott, thanks for mentioning my blog in the related links section. It's honor to be mentioned here in your great blog.

    Cheers,
    Rahul

    ReplyDelete
  5. Thanks for this post.Very important and timely article. Information provided is concise and informative. Keep up the great work!
    file naming conventions



    ReplyDelete