About keying: color keys, luminance keys, and difference

keys
Keying is defining transparency by a particular color value or
luminance value in an image. When you key out a value, all pixels that
have colors or luminance values similar to that value become
transparent.
Keying makes it easy to replace a background, which is especially
useful when you work with objects too complex to mask easily. When
you place a keyed layer over another layer, the result forms a
composite, in which the background is visible wherever the keyed layer
is transparent.
You often see composites made with keying techniques in movies, for
example, when an actor appears to dangle from a helicopter or float in
outer space. To create this effect, the actor is filmed in an appropriate
position against a solid-color background screen. The background color
is then keyed out and the scene with the actor is composited over a new
background.
The technique of keying out a background of a consistent color is often
called bluescreening orgreenscreening, although you dont have to use
a blue or green screen; you can use any solid color for a background.
Red screens are often used for shooting non-human objects, such as
miniature models of cars and space ships. Magenta screens have been
used for keying work in some feature films renowned for their visual
effects. Other common terms for this kind of keying are color
keyingand chroma keying.
Difference keying works differently from color keying. Difference
keying defines transparency with respect to a particular baseline
background image. Instead of keying out a single-color screen, you can
key out an arbitrary background. To use difference keying, you must
have at least one frame that contains only the background; other frames
are compared to this frame, and the background pixels are made
transparent, leaving the foreground objects. Noise, grain, and other
subtle variations can make difference keying very difficult to use in
practice.
Keying effects, including Keylight
After Effects includes several built-in keying effects, as well as the
Academy Award-winning Keylight effect, which excels at
professional-quality color keying. (See Keying effects and Matte
effects.)
For information on the Keylight effect, see its documentation in the
folder in which the Keylight plug-in is installed, or on the Foundry
website.
Keylight is not included with the trial version of After Effects CS5.
(See Third-party plug-ins included with After Effects.) Keylight is
included in the trial version of After Effects CS5.5 and later.
Note:
Though the color keying effects built into After Effects can be useful for
some purposes, you should try keying with Keylight before attempting
to use these built-in keying effects. Some keying effectssuch as the
Color Key effect and the Luma Key effecthave been superseded by
more modern effects like Keylight.
For a video tutorial on keying with Keylight, go to the Adobe website.
Mark Christiansen provides tips and techniques for using Keylight in
an excerpt from his book After Effects Studio Techniques: Visual
Effects and Compositing on the Peachpit Press website. In an excerpt
from the Color Keying in After Effects chapter of After Effects
Studio Techniques, Mark Christiansen provides detailed tips and
techniques for color keying, including advice on which keying effects
to avoid and how to overcome common keying challenges.
For a step-by-step tutorial demonstrating the use of the Color
Difference Key effect, the Matte Choker effect, the Spill Suppressor
effect, and garbage masks, see the Keying in After Effects chapter of
the After Effects Classroom in a Book on the Peachpit Press website.
Jeff Foster provides free sample chapters from his book The Green
Screen Handbook: Real World Production Techniques. The sample
chapters cover basic compositing, color keying, garbage mattes, hold-
out mattes, and how to avoid common problems with greenscreen
shots. For more information, see the Adobe website.
Rich Young collects more tips and resources for keying on his After
Effects Portal website.
Tips on color keying and compositing from experienced compositor,
Chris Zwar.
Chris & Trish Meyer shares tips & resources for color keying with
Keylight and other effects.
Note:
Keep in mind that generating a high-quality key can require the
application of multiple keying effects in sequence and careful
modification of their properties, especially if the footage was shot
without considering the requirements of the compositor.
Shooting and acquiring footage for keying
Keying footage that was not acquired correctly is much more difficult
than is keying footage that was acquired with keying in mind. For best
results, use Adobe OnLocation to monitor color and lighting as you
acquire footage for color keying.
For tips on shooting footage so that color keying is easier and more
successful, see Jonas Hummelstrands General Specialist website.
Alex Lindsay provides an article on the ProVideo Coalition
website about setting up and lighting a green background in preparation
for color keying work. In part two of his greenscreen primer on
the ProVideo Coalition website, Alex Lindsay provides detailed
information about cameras, codecs, and capture systems to use when
acquiring footage for color keying.
Light your color screen uniformly, and keep it
free of wrinkles.
Start with the highest-quality materials you
can gather, such as film that you scan and
digitize.
Use uncompressed footage (or, at least, files
with the least possible amount of
compression). Many compression algorithms,
especially the algorithms used in DV, HDV,
and Motion JPEG, discard subtle variations in
bluewhich may be necessary to create a
good key from a bluescreen. Use footage with
the least color subsampling possiblefor
example, 4:2:2 rather than 4:2:0 or 4:1:1. (For
information about color subsampling, see
the Wikipedia website and the Adobe
website.)
Robbie Carman and Richard Harrington provide an excerpt on
the Peachpit website from their book Video Made On A Mac that
demonstrates how to plan, shoot, key, and composite a greenscreen
shot.
Tips for keying with After Effects
Noise and compression artifacts can cause
problems for keying, especially difference
keying. Often, applying a slight blur before
keying can reduce noise and compression
artifacts enough to improve keying results. For
example, blurring the blue channel for DV
footage can smooth out noise in a bluescreen.
The KeyerforDV animation preset available
through the AE Enhancers forum automates
the process of blurring the blue and green
channels before keying DV footage.
Use a garbage matte to roughly outline your
subject so that you dont have to waste time
keying out parts of the background far from
the foreground subject. (See Use a garbage
matte.)
Use a hold-out matte to roughly protect areas
that are of a similar color to the background
from being keyed out. (See Use a hold-out
matte.)
To help you view transparency, temporarily
change the background color of the
composition, or include a background layer
behind the layer you are keying out. As you
apply the keying effect to the layer in the
foreground, the composition background (or a
background layer) shows through, making it
easy to view transparent areas.
(See Composition settings.)
For evenly lit footage, adjust keying controls
on only one frame. Choose the most intricate
frame of the scene, one involving fine detail
such as hair and transparent or semitransparent
objects, such as smoke or glass. If the lighting
is constant, the same settings you apply to the
first frame are applied to all subsequent
frames. If lighting changes, you may need to
adjust keying controls for other frames. Place
keyframes for the first set of keying properties
at the start of the scene. If you are setting
keyframes for one property only, use Linear
interpolation. For footage that requires
keyframes for multiple interacting properties,
use Hold interpolation. If you set keyframes
for keying properties, you may want to check
the results frame by frame. Intermediate
keying values may appear, producing
unexpected results.
To key well-lit footage shot against a color
screen, start with the Color Difference Key.
Add the Spill Suppressor to remove traces of
the key color, and then use one or more of the
other Matte effects, if necessary. If you are not
satisfied with the results, try starting again
with the Linear Color Key.
To key well-lit footage shot against multiple
colors or unevenly lit footage shot against a
bluescreen or greenscreen, start with the Color
Range key. Add the Spill Suppressor and other
effects to refine the matte. If you are not
completely satisfied with the results, try
starting with or adding the Linear Color Key.
To key dark areas or shadows, use the Extract
Key on the Luminance channel.
To make a static background scene
transparent, use the Difference Matte Key.
Add the Simple Choker and other effects as
needed to refine the matte.
After you have used a key to create
transparency, use Matte effects to remove
traces of key color and create clean edges.
Blurring the alpha channel after keying can
soften the edges of the matte, which can
improve compositing results.
To the top
Use a garbage matte
A garbage matte (or junk matte) removes unneeded portions of the
scene, resulting in a rough area that contains only the subject that you
want to keep. When you are working with a poorly lit or uneven color
screen (for example, a bluescreen or greenscreen), sketching a garbage
matte around the subject can greatly reduce the amount of work that
you have to do in keying out the background. However, if you spend a
lot of time making a perfect garbage matte that exactly outlines the
subjectessentially rotoscopingyou lose the time-saving advantage
of keying.
1. Create a mask to roughly outline a subject.
2. Apply one or more keying effects to mask out the remainder of
the background.
3. Apply Matte effects as necessary to fine-tune the matte.
Aharon Rabinowitz provides a video tutorial on the Creative COW
website that shows how to create a super-tight garbage matte using
Auto-trace.
To the top
Use a hold-out matte
Use a hold-out matte (also known as a hold-back matte) to patch a
scene to which a keying effect has been applied.
A hold-out matte is a masked-out portion of a duplicate of a layer that
you have keyed. The duplicate is masked to include only the area of the
image that contains the key color that you want to preserve as opaque.
The hold-out matte is then placed directly on top of the keyed layer.


Example of using a hold-out matte
A. Original bluescreen image. The background for the number is also blue. B. After
keying, the background for the number is also transparent. C. Hold-out matte containing
the part of the image you want to remain opaque D. When the hold-out matte is placed on
top of the keyed image, the background for the number is now opaque.
1. Duplicate the layer containing the color screen.
2. Apply keying effects and Matte effects to the original layer to
create transparency.
3. On the duplicate layer, create masks to mask out everything in
the image except the area that you want to preserve.
4. Make sure that the copy (the hold-out matte) is positioned
directly on top of the keyed layer.