Hello, I’m Victoria Nece. I’m a documentary animator, and today I’m going to show you how to use your Kinect to animate a digital puppet like this one in After Effects.

If you have a Kinect that came with your Xbox, the first thing you’re going to need to do is buy an adapter so you can plug it into your computer’s USB port. You don’t need to get the official Microsoft one — I got a knockoff version from Amazon for six bucks and it’s working just fine.

Next you’re going to need to install a ton of different software. It’s all free and open-source, but I’m warning you now: there’s a lot of it. Check the links below to see what you need for your particular setup.

Here’s a quick overview of how it’s all going to work. Once you’re up and running, you’re going to be using a Processing app called KinectToPin, written by the very talented animator Nick Fox-Gieg. That’s where you actually capture the tracking data, as well as where you convert it to keyframe information After Effects can understand.

Then on the After Effects side of things, you’ll set up a skeletal rig for a layered 2D puppet and apply the tracking data to bring it to life.

It’s not an easy process, but the results are worth it.

Required Software:

• OpenNI + PrimeSense Sensor Module (You need both): http://www.openni.org/Downloads.aspx
• OSCeleton: https://github.com/Sensebloom/OSCeleton
• SimpleOpenNI: http://code.google.com/p/simple-openni/
• Processing: http://processing.org/
• KinectToPin: https://github.com/N1ckFG/KinectToPin
• After Effects CS3+ (CS5+ to use tutorial files)

Special Mac note: works in Snow Leopard, may not work in Lion.

If you need a refresher on how exactly this (admittedly rather baffling at first glance) plugin works, take a look at my tutorial.

Click here to view the embedded video.

We’ve been featured on Urlesque and The Daily What. It’s all very exciting.

Here’s how it looks at the moment:

Here’s the underlying vector map layer:

And even that map began life as a set of even more boring public domain vectors. Everything else is AE-generated. (And yes, the size of Hawaii and Midway are exaggerated slightly for increased visibility on your television screen. Shhhh.)

Nothing is finalized yet; the map that actually ends up in the film could be more different still.

Note:
Almost a year to the date I originally wrote this tutorial, I can finally make it public because the film has been released! You can watch it here: Jury Selection: Edmonson v. Leesville Concrete Company.

This is the trailer, which is almost entirely done with the effect I’m about to explain:

Click here to view the embedded video.

I’m currently working on a new Constitution Project film about Edmonson V. Leesville Concrete Company. In the past, we’ve used methods like digital puppetry to avoid filming reenactments. Edmonson is a much more recent court case than the others we’ve covered (1991), but the US Supreme Court only permits audio recordings of oral arguments, so there’s still no footage of the proceedings.

We were actually able to interview several of the people involved, however, resulting in tons of sharp, clear green screen footage (It’s also our first CP film in HD), as well as hundreds if not thousands of still images. So what to do with them?

Enter a new feature in After Effects CS4: the Cartoon effect. We’re going to take those photos and turn them into something like this:

Preparing The Images

1. Choose the footage clip or set of photos you want to use. If you’re hoping to reposition the person within the frame, make sure you’re not using footage or stills in which parts of their body you want visible are cropped out. Don’t import stills as a sequence.

2. Drop the footage or images into a new comp with a name like “stills precomp”. (We’re working at 1920×1080 HD, 29.97fps.) Select all and adjust the sizing and position so the portion of the image you want visible is within the comp’s borders.

3. If you’re using stills, set their duration to the length you want each to last, select them from first to last (this is important!) and choose Animation -> Keyframe Assistant -> Sequence Layers…

You can choose to overlap them or not — this gives a nice faded result, and allows you to stretch fewer stills over a longer period of time, but if there are lots of stills it’s not really necessary.

Adjust any of the images that seem to be particularly out of line position-wise.

4. Nest the image sequence comp within a new comp and play it through. If the sequence is particularly shaky it might be worth using the motion tracker for stabilization. Mask out any areas that contain background junk.

5. Pre-compose again (Ctrl+Shift+C/Cmd+Shift+C) and enable “collapse transformations” — the little sun icon on the layer switches.

Adding The Effects

6. We need to key out the green chromakey background. Keylight is the best filter for this, but it often leaves a slightly fuzzy matte unless you tweak the clip black and clip white settings. Since we don’t need to be super-precise, you can cheat this by adding a quick Simple Choker effect on top instead. I created a preset for this project because all of our photos are from the same source, but you may need to adjust the numbers for your own materials.

7. Create a new adjustment layer called “Cartoon” (Layer -> New -> Adjustment Layer).

8. Apply the Cartoon effect, and make sure to set render to “Edges.” Adjust the sliders until you get a result you like. I created another animation preset for this step, but it still has to be tweaked for each shot to keep everything consistent — the Edge Threshold and Width sliders in particular need to be changed depending on the source image.

9. Add a Drop Shadow effect. Size to taste.

You now have a cartooned cutout of your source. What you do with it is pretty much up to you. For our project we want to composite these stylized photo sequences over a range of different backgrounds, everything from simple textures to documents.

Adding Some Texture

10A. The Simple Route: Drop the texture file into the same comp. Experiment with different blend modes until you find one that looks good with your materials. Add a vignette* to burn the edges. Animate the texture (perhaps adjust the scale slightly?) to give the scene a little more motion. Here’s an example of a simple texture application:

10B. The Slightly More Complex Route: Drop the texture file into the comp, and set the blend mode to Silhouette Luma. Add a vignette adjustment layer. Precompose everything, and add the texture again to the new comp — below the cutout. Animate as needed. You may want to set keyframes in both the final comp and the precomp, depending. Add another vignette if necessary.

Final result, with a different texture:

*I have a standard preset for vignettes since I use them so much, but the quickest way to make one from scratch is to create a new adjustment layer, double-click on the rounded rectangle mask tool (which gives you a mask the size of the layer), set the mask to “subtract” with a large feather value, then drop a Levels filter on the layer and push the blacks way up.

That’s about it! If you’re going to be doing a lot of these, presets for the key, cartoon effect and vignette are definitely worth making.

Here’s a still from the raw footage (DVCPROHD 1080i60):

And here’s the after shot, with only four very basic After Effects filters applied (click to embiggen):

I used Leave Color to keep just the blues and desaturate the rest of the image, then boosted the saturation a bit to make the remaining color really pop. After that it was just your basic Levels and a high-contrast instance of Curves.

Oh, and one extra layer on top as a vignette — a black solid with a rounded rectangle mask (double-click the mask tool to get one that fills the layer) set to subtract and heavily feathered. That’s all!

Leave Color has simple parameters but dramatic results, and is especially good as a quick trick if you need to draw a viewer’s eye to a key element of a scene. All you have to do is decide which color you want to keep. For example, say you have a shot of a bunch of people at a party, with one woman in a bright red dress. Choose that red with the eyedropper, then slide “Amount to Decolor” to 100%. Only the color of the dress will remain, and she instantly becomes the center of attention.

Sin City and Pleasantville (interesting combination of titles when you think about it…) are both good examples of how visually arresting color highlights can be on a black and white scene.

And did I mention it’s free? You can download it here.

It’s also really, really intimidating at first glance. It has more than eighty different keyframable parameters, and almost all of them have extremely vague names like “Xform 2 Var 6 bent” or “spatial oversample factor.” It’s also slow to render, sparsely documented and more than happy to crash on a regular basis.

Eep.

So, like, how do you make it… do stuff?

It’s not quite as scary as it looks.

BASIC STEPS

1. Make sure the AEFlame plugin is installed and the presets file is hanging out in your C:/ directory before launching After Effects.

2. Create a new composition.

I’m working in NTSC D1 widescreen 720×486. Create a new comp-size solid — color doesn’t matter.

3. Choose Render -> AEFlame from the Filters menu.

Look, fractals! You may want to work at draft quality and quarter resolution until you’re sure of your design — it helps to cut down on the crashes. (You can set AEFlame itself to draft quality through the Global Commands drop-down menu as well.)

KEY PARAMETERS

4. And now for the secret of AEFlame: one little slider changes everything. I think you’d see this plugin in much wider use if this option was off on its own with a big “USE THIS ONE” label and a friendly red arrow pointing to it. But it’s not. So instead, you need to look for “Parameter Set,” the first option under Global Parameters.

You can put any number you want in here, but it needs to be between 0 and 300 for anything to happen.

Scroll through your options until you get to a good starting point. I’ve picked 242, for argument’s sake. It’s pretty:

5. Choose a set of colors for your fractal by scrolling through the “Colormap #” option (also under Global Parameters). There are 83 to pick from. Number 36 looks good for now.

Adding a couple keyframes to the Colormap Rotation parameter can give you an instant color-shifting background pattern, and is an easy way to create something that’s not too distracting.

ANIMATION

Now that you’ve created a fractal, it’s time to add some motion. There are several different approaches you can take at this point, but all of them require at least a little bit of digging into the Scary Confusing Other Menus that we have heretofore avoided:

…Yikes.

Fortunately, AEFlame has automated the process for us a bit. I’ll get to that in a minute, but first, some attempted explanation of what all this actually means:

It’s easiest to think of each Xform as an “influencer” — play with the numbers, and you’ll change the shape of the fractal. We’re going to stick with Xform 0 and Xform 1 for this exercise. (Xform 2 and 3 are disabled by default. You can enable them if you want to experiment, but they tend to slow processing down and you get perfectly good results without them.)

The first two options are Density, which affects how strong an influence each Xform menu has over the fractal as a whole, and Color, which decides how much power that menu has over the color mapping. I tend to leave them alone.

The remaining parameters fall into two categories: vars and coefficients. As best I can figure, there are multiple ways of drawing and distorting a fractal, and the var controls decide how much of each to use. Linear creates straight lines, sinusoidal waves, spherical circles, swirl, well, swirls… you get the idea. The coefficient numbers get added into the mix as multipliers. The fact is, all you really need to know is that fiddling with the numbers will give you different shapes.

So. On we go.

6. You can tweak all 30 or so of these parameters individually, but AEFlame is happy to do it for you! Set keyframes for all the vars and coefficients in Xform 0 and 1, then move a few seconds down the timeline.

7. Choose Insert Keys from the Global Commands menu to automatically create keyframes for every parameter you’re animating.

This saves a surprising amount of time.

8. Use one of the options on the Global Randomize menu to randomize everything at once, or the individual Xform menus to randomize one section at a time. There are lots of different choices, and they vary in intensity. Your fractals can morph into something wildly different, or evolve smoothly and gently. And if you don’t like the result of randomizing, just click it again!

(Choosing “Vector all coeffs forward” (left) tends to produce a nice subtle effect, whereas it’s a pretty good bet that “Randomize All” (right) is going to give you something that looks nothing like what you started with.)

9. Experiment until you get something you like, then adjust the keyframes to control timing — some animations really need ease in/ease out to look natural.

FINE TUNING

10. So now you’ve made a cool shape and have it swirling around in space. Time to add a final bit of polish and make your work smooth and shiny and ready to render.

Before you begin this step, SAVE YOUR WORK. Messing with AEFlame’s quality settings is a good way to crash After Effects, especially when you’re working with high-res files. Experiment with care and try to keep these settings as low as possible.

There are several different ways to adjust the image quality. The simplest method is to choose an option from the Global Commands menu. Do you need to go all the way up to “Ultra High Quality”? Probably not. High is probably enough, and a better compromise between smoothness and render time. (Ultra High could easily be relabeled Ultra Slow.)

If you want to adjust quality manually, there are three sliders in particular that you need to pay attention to, all located next to each other on in the Global Parameters menu. They should seem quite familiar if you’re used to rendering in 3D software:

A) Sample Density. Think of this as an “overall quality” setting — lower values give you grainy results, higher values give you beautiful swirled images but will likely crash your PC. Cranking it all the way up to 200 will give you beautiful, virtually grain-free results, but I find values around 30 still render with reasonable speed and look quite good.

B) Spatial Oversample Factor. This performs a function similar to anti-aliasing, smoothing edges and getting rid of jaggy bits. It’s very processor-intensive, as it essentially requires re-rendering areas multiple times in order to smooth them (someone please correct me if I’ve explained this wrong). You don’t need a large value here — even choosing Ultra High Quality only sets oversampling to 10 out of a possible 20. If the Sample Density slider is set too low, a high oversample value will darken your image.

C) Spatial Filter Radius. This increases the antialiased look by blurring the resulting image. Keep it set low — the default values are around .5, and you won’t need much more.

For comparison’s sake, check out the following (click for big):

Draft Quality (Spatial Oversample: 1, Spatial Filter Radius: 0, Sample Density 1. Render time: less than a second.):

Good Quality (Spatial Oversample: 2, Spatial Filter Radius: 0.5, Sample Density 50. Render time: 13 seconds.):

Ridiculous Quality (Spatial Oversample: 4, Spatial Filter Radius: 0.5, Sample Density 200. Render time: 52 seconds.):

The difference in quality between #2 and #3 just isn’t worth the difference in render time.

MOVING BEYOND AEFLAME

You can stop animating here, render out your fractal sequence and be done for the day. But AEFlame produces even more interesting results when combined with other features of After Effects. (Although it’s a good idea to pre-render the flame layer once you’re satisfied with it. Your subsequent experiments will be faster and less prone to crashing.)

Here are a few things to try to make your flame fractals more unique:

• Composite your AEflame layer over a background — even a simple gradient ramp can be very effective, and choosing various blend modes will allow you to create many different looks from a single fractal.
• Add additional filters. CC Time Blend can do stunning things when given AEFlame as input. The “composite under” setting generates animations with a soft, dreamlike quality. Time Blend can be slow, but it’s a great deal faster than AEFlame at Ultra High Quality. Glow and radial blur also produce some nice effects.
• Create multiple flame layers with different parameter sets, set their blend modes to “add” and composite them in 3D so they swirl around each other.
• Use some basic color correction techniques to make things pop. Levels and Curves are always useful, and duplicating a layer, blurring it slightly then setting it to “Overlay” will give you tons of glowy contrast.
• Write a bit of code. AEFlame handles expressions well. I find I get nice results when I map various parameters to an audio clip. You can do this either by using a plugin like Trapcode SoundKeys or converting your music to keyframes (Animation -> Keyframe Assistant -> Convert Audio to Keyframes) and mapping things to the amplitude. Check out Dan Ebberts’ fantastic guide to expressions at MotionScript.com — he has some helpful tutorials on how to do it. Mapping a waveform to brightness and scale creates a nice pulsing effect.

And, most importantly, play around! You have approximately one meelyun parameters here to experiment with — see what they do! The best things I’ve created with AEFlame started out as complete accidents.

So there you have it. Not SO scary after all, is it? And with so many options at your disposal, you’ll never create the same fractal twice!