Yes, it’s the part you’ve all been waiting for: it’s finally time to set up your character layers!

This was originally going to be in Part 3, but there are so many steps I realized I needed to break things down a bit more.

About the Puppet Tool

We’re going to be using the Puppet Tool a little differently than it was originally intended to be used. It was designed for a single layer puppet, but when you try to animate a character that way it gives you a very wiggly result — what I refer to as the “Bendy Elbows” problem. It’s great for deforming or reshaping a layer, or making old-timey cartoons, but it won’t give you anything close to realistic movement.

...Yikes.

And if you try using all 15 points of your KinectToPin tracking data with a flat puppet the result is.. unfortunate.

(I don't think legs should bend like that.)

However, if you only have two puppet pins on a layer, you get a completely different effect: instead of making it go all wibbly, they work sort of like multiple anchor points with the layer stretched between them.

So that’s why we’re putting the source data on one layer and using expressions to link to it to control separate pieces.

Designing Your Character for Kinect Control

As if this process wasn’t complicated enough already, it also makes building your character a little more work: instead of just creating a single layer, you need to build your puppet out of lots of little pieces so that it can rotate wherever there’s a joint — you need a hand, a forearm, an upper arm and so forth.

One thing is a bit easier though. Instead of having to use the Puppet Tool’s layer overlap feature, you can change what part of your character is in front by simply changing the layer order. This also means that if you have an element that needs to be in front of some layers and behind others, like the head of a character with long hair, you need to make it two separate layers.

You can use your own preferred technique to create your puppet, whether it’s drawn or cut up in Photoshop, created in Illustrator or even generated via After Effects shape layers like this one. I’ve used this process with everything from Illustrator vectors to pieces cut out of vintage photos and artwork, but for this demo I used shape layers so I could give out a single .aep file with no dependencies. You can’t just cut up a flat puppet or photo, unfortunately. You need to give each layer a little bit of overlap with the ones it connects to so the puppet doesn’t come apart at the joints.

Also, if you’re importing your layers from another program, make sure to crop them to the layer dimensions on import (not the comp’s) so you aren’t dealing with unwieldy giant rectangles the whole time.

Scaling: Getting it Right Before You Rig

Regardless of your design technique, be sure the layers are decently high resolution so you have some flexibility in how close you can zoom in on them. Even if your character is made of vectors, you’re going to lose some sizing flexibility when you add the Puppet pins. Here’s why: when you first add pins, the Puppet Tool works by creating a mesh of your layer at that layer’s original resolution. And you can’t use Collapse Transformations on a layer with a Puppet Tool instance — it breaks everything — so you want each piece as big as possible at this point. All the scaling down needs to happen inside the puppet animation comp, and all the scaling up needs to happen once that comp is nested elsewhere. Don’t worry if the raw elements are actually bigger than your animation comp itself: even though the puppet comp is only 640×480, you can scale back up into HD later as long as you plan things right at the start, because you can apply Collapse Transformations to the puppet comp once it’s nested in a larger comp.

In short: make sure all your layers are nice and big at the point the Puppet Tool gets to them.

If you’re using vector or shape layers, you need to precompose them before you apply the puppet tool. Same goes for if you have a layer that’s going to contain animation inside itself, such as lip sync on a head. Make your precomp the size of the layer at 100% scale, not the size of the puppet comp, or you’ll lose the ability to enlarge it later.

Setting Up After Effects for Rigging

Okay, so, you’ve got everything loaded in, precomposed, and the right scale? Good. Now you can connect it to the template you built in Part 3.

You’re going to be linking Puppet pins to the expression point controls in the template. The easiest way to do this is to open up multiple Effects palettes for the different control layers down the side of the screen. You can do this by clicking the lock icon in the Effects window, then opening the next one below it, locking that one, and so on. Having all the points visible makes it easy to pickwhip the right one.

Connecting to Control Layers

Let’s start with the left lower leg. I tend to hide all the layers I’m not currently rigging so I don’t get confused. Use the Puppet Tool to create a pin where you want the ankle pivot to be, then one where you want the knee pivot to be.

When you created these pins, the Puppet Tool also created the mesh it’s going to use to deform the layer, and you can see it by clicking the “Show Mesh” checkbox. If you want to make sure the edges of your limb don’t get cropped off, you can set the mesh expansion up to five or so. If things seem really slow, you can drop the number of triangles way down and that should help, but if you have too few triangles things won’t curve smoothly.

Now, hit U to show the keyframes on the timeline.  For Pin 1, alt-click the stopwatch, then type fromComp( [pickwhip leftAnkle control point from leftLeg layer] ). The “fromComp” transform makes the Puppet Tool understand where to put the pins relative to the layer’s dimensions instead of to the comp as a whole.

The expression should look like this:

fromComp(thisComp.layer("leftLeg").effect("leftAnkle")("Point"))

Now, do the same thing for Pin 2, but link it to the left knee:

fromComp(thisComp.layer("leftLeg").effect("leftKnee")("Point"))

Now, scale down the layer until it appears to be a sensible size and proportion.

If you built your character all at once, you’re going to scale down all the layers to this size. Try scrubbing through the timeline. Congratulations, you’ve successfully animated a shin!

You’re going to follow the same basic pattern for the upper and lower arm and leg pieces: add a fromComp layer space transform, then choose the relevant point control. The torso, hands, feet and head work differently, though, so save them for last.

So: left upper leg. Point on the knee, point on the hip joint.

The knee pin gets the same expression it had on the lower leg layer. The hip joint gets:

fromComp(thisComp.layer("leftLeg").effect("leftHip")("Point"))

Hey check it out, you have a leg!

Same deal on the right side of the body. Add the right ankle:

fromComp(thisComp.layer("rightLeg").effect("rightAnkle")("Point"))

And the knee point:

fromComp(thisComp.layer("rightLeg").effect("rightKnee")("Point"))

Now add the knee point to the upper leg:

fromComp(thisComp.layer("rightLeg").effect("rightKnee")("Point")) 

And the hip point:

fromComp(thisComp.layer("rightLeg").effect("rightHip")("Point"))

Starting to make sense now?

If it looks like the legs are bending backwards or on the wrong side, check or uncheck the Invert Pins box.

Now we’ll do the arms. For the forearm, add pins to the right wrist:

fromComp(thisComp.layer("rightArm").effect("rightWrist")("Point"))

And the right elbow:

fromComp(thisComp.layer("rightArm").effect("rightElbow")("Point"))

And for the upper arm, you need the right elbow:

fromComp(thisComp.layer("rightArm").effect("rightElbow")("Point"))

And the right shoulder:

fromComp(thisComp.layer("rightArm").effect("rightShoulder")("Point"))

That’s the right arm done. Now for the left.

Left forearm: left wrist:

fromComp(thisComp.layer("leftArm").effect("leftWrist")("Point"))

And left elbow:

fromComp(thisComp.layer("leftArm").effect("leftElbow")("Point"))

Left upper arm is the left elbow:

fromComp(thisComp.layer("leftArm").effect("leftElbow")("Point"))

And the left shoulder:

fromComp(thisComp.layer("leftArm").effect("leftShoulder")("Point"))

Scrub through: it’s starting to come to life!

Next we’re going to rig the torso, which is a little more complicated.

We’re actually okay with it bending and flexing, so it gets five points: the hip and shoulder pins at the corners, as well as the waist pin right in the middle. Connect them the same way you linked up the limbs. Here are the five separate expressions:

fromComp(thisComp.layer("Spine").effect("Waist")("Point")) 

fromComp(thisComp.layer("leftLeg").effect("leftHip")("Point")) 

fromComp(thisComp.layer("rightLeg").effect("rightHip")("Point")) 

fromComp(thisComp.layer("rightArm").effect("rightShoulder")("Point")) 

fromComp(thisComp.layer("leftArm").effect("leftShoulder")("Point"))

Scale that down, too:

You could also add the neck pin if you wanted, but whether that works well really depends on the design of your character.

Adjusting Pin Placement

Now, take a look at how things are moving. If it seems like a shoulder pin is in the wrong place, you can delete it and create a new pin in a different spot. The Puppet Tool will keep using the same underlying mesh. Copy the expression before you delete the pin so you can add it to the new one right away. One thing that’s really important to point out here is that if you make any changes to the shape of the layer itself, you need to delete more than just the puppet pins — you need to delete the entire puppet effect instance. Otherwise, it’s going to keep on using the old, outdated layer mesh. This is also why you can’t copy and paste my puppet pins to your own character.

The Hands and Feet

Next we’re going to attach the hands and feet. These are quick and easy: you don’t need the puppet tool for these, and you don’t need the fromComp layer space transform. However, instead of a pin, you’re going to apply it to the layer’s position transform. Just pickwhip the right arm control, like so:

thisComp.layer("rightArm").effect("rightWrist")("Point")

Now, move the anchor point until the hand is connected to the lower arm in the right spot. The hands and feet are the only things you’ll actually need to animate by hand to match the Kinect-captured movements. Just set rotation keyframes so they’re in the correct position. If you download my sample puppet, you may notice I have Time Remapping enabled on the hands as well. Stay tuned for Part 5 to learn how to use that to change the hand pose.

Now, do the same on the left side:

thisComp.layer("leftArm").effect("leftWrist")("Point")

And now the feet. Right:

thisComp.layer("rightLeg").effect("rightAnkle")("Point") 

And left:

thisComp.layer("leftLeg").effect("leftAnkle")("Point")

Rigging the Head: Two Options

The last thing we’re going to rig is the head. You have a couple of choices at this point. If you want to animate head tilt yourself, or have a head that’s going to change shape from talking or varying facial expressions, you should attach it the same way you did the arm and leg by pushing the head’s anchor point to the bottom and linking its position transform to the neck point control:

thisComp.layer("Spine").effect("Neck")("Point")

Scale to match:

If you want to use the puppet tool instead, add one point at the neck and one in the center of the face.

I find the head data is the most unreliably tracked, so I’ve split the x and y positions up so I can offset them individually if need be. In this case, here’s the expression I used for the neck in order to move the Y point 21 pixels:

x = thisComp.layer("Spine").effect("Neck")("Point")[0];
y = thisComp.layer("Spine").effect("Neck")("Point")[1]-21;
fromComp([x,y])

Don’t forget the fromComp transform at the end!

And here’s the one I used on the head point:

x = thisComp.layer("Spine").effect("Head")("Point")[0]+2;
y = thisComp.layer("Spine").effect("Head")("Point")[1]-3;
fromComp([x,y]) 

Without offsets:

With offsets:

There’s no magic to the offset numbers I used… they’re just the numbers I found worked through trial and error when applied to this one particular track. Stretching a face even a pixel or two is noticeable in a way stretching a forearm isn’t, so you just kind of have to eyeball it.

Results

You’re all set! Hit play and check out the results, or to see it nice and big drop it into an HD comp and enable Collapse Transformations, like so:

You now have a fully rigged Kinect-controlled puppet, and all you have to do to change its actions is to paste new tracking data to the source layer.

After a little more tweaking, here’s the final result:

Stay tuned for Part 5 for finishing techniques, including a more advanced way to animate the hands and feet, a quick and dirty trick to automate lip sync, and how to bring your puppet into HD.

Welcome back! So now that you have your tracking data recorded, it’s time to build a character to apply it to. This is where things get a bit complicated.

Template Setup

We’re going to start by building a reusable template that you can use as the basis for any Kinect-controlled puppet. If you want to skip this section and download your own version of the template to start with, go for it, but I think knowing how it all works actually makes rigging easier.

1. Create a 640×480 24fps comp

Start by creating a new comp. Let’s call it “Puppet Template.” It has to match the dimensions and framerate of the Kinect, so: 640 by 480, 24 frames per second. Don’t worry, you’ll still be able to composite your puppet in HD later on.

I’ve added a reference image to help explain what’s going on: the Kinect tracking pins are numbered 1 to 15, starting with the left foot and ending with the head.

2. The Source layer

Now, we’re going to set up the layer that will hold the tracking data. Create a null or a new solid, and label it “Source.” It also needs to be 640×480. You can hit Ctrl + Shift + Y to resize it.

Select the puppet tool, and add 15 points to this layer, in roughly the shape of the Kinect source data.

They don’t need to be perfect, but you can use the reference image as a template if you want. Now, you aren’t going to be animating this layer directly: these pins are where you’re going to paste your tracking data so the puppet pieces can access it. That is, you’re going to use them as a source of information to control another set of pins on different layers.

It’s good to have a sample track to test your rig with.

To add your mocap data, open up its text file, copy the whole file, then select all the pins and hit paste.

Kinect tracking data can be very choppy, so we need to smooth it out, or all of our characters will look as if they’ve had far too much coffee. After Effects actually has a nondestructive smoothing function built into the program, but it’s only accessible as an expression. I’ve been using this software for more than a decade now and had no idea it was there, so I need to thank Roland Kahlenberg on CreativeCow for pointing me in the right direction on this, as it’s what really makes the process workable.

So let’s smooth things out a bit. Head down to the timeline, and hit U to show all keyframes. Hold alt and click the stopwatch to create an expression. It defaults to self-referencing, so that snippet of script just means “use these keyframes here to control this point.” You’re not going to delete that bit, just add a little piece of code on the end that will make the Kinect data play nice. Type .smooth(.5,15) .

You have to do this for all 15 pins, but you only have to do it this once.

You can take a look at the smoothed out path by clicking “show post expression graph” then switching to the Graph Editor.

You can really see the difference it makes! The smooth method works by averaging out the motion over time. The first number is the amount of time averaged — .5 means half a second. And the second value is the number of samples over that time. Make sure it’s an odd number so that the frame you’re looking at gets factored in. The bigger the number, the smoother the motion, but it may slow down your machine, and if you average things too much you’ll start to get into creepy mocap jello territory. I find .5 and 15 work pretty well for me, but you can adjust the numbers to suit your own designs.

3. Adding Invert Pins Option

Now, sometimes the data comes in with left and right inverted. The next thing we’re going to build gives us a quick method of flipping the point sources so that when that happens you just have to check a box instead of rewriting all your expressions.

Create a new layer, and call it Invert Pins. Add a Checkbox expression control (Effect -> Expression Controls -> Checkbox Control).

You can rename that if you want. I’m calling mine “Invert Pins?”.

We’re going to write conditional expressions that refer to it — that means if this box is checked, they’ll get the tracking data from one side, and if it’s unchecked, they’ll get it from the other.

4. Creating control layers

The next step is to create the control layers for the different body parts. You could just have all the expressions on one layer, but separating them gives you a little flexibility and makes it easier to find the body part you’re looking for when you’re connecting the pins on your rig to the controls.

Start by creating another null. Set the anchor point to 0 by selecting the layer, pressing a, then typing 0. If you need to shift a whole arm or leg over once you have things rigged, you’ll be able do it by moving these layers.

Let’s start with the left leg. Rename the layer leftLeg.

Create an expression point control (Effect -> Expression Controls -> Point Control), and call it leftAnkle.

The next bit is easier if you hit U to show all the points on your source layer so you can grab them with the pickwhip.

Enable expressions, and clear out the default one. Now it’s time for that conditional expression I mentioned earlier. We’re setting this layer up to change its content based on the Invert Pins checkbox.

Here’s how to add the expression. The bits in bold you should type exactly as they are, the bits in italics are instructions, and the comments after the //s are just to help explain what’s going on if you’re new to conditional expressions.

if( [Pickwhip the Invert Pins checkbox control] == false) // That’s saying “if the checkbox isn’t checked”…
{
fromWorld( [Pickwhip Source pin 1’s position] ) // …grab the position of the left ankle pin and convert its coordinates to World layer space…
}else{        // …otherwise…
fromworld( [Either pickwhip pin 4’s position, or just copy and paste the pin 1 info and change the 1 to a 4] ) //  …get the position of the other ankle in World layer space and use that instead.
}

The final expression should look like this:

if( thisComp.layer("InvertPins").effect("Invert Pins?")("Checkbox") == false) {
 fromWorld(thisComp.layer("Source").effect("Puppet").arap.mesh("Mesh 1").deform("Puppet Pin 1").position)
}else{
 fromWorld(thisComp.layer("Source").effect("Puppet").arap.mesh("Mesh 1").deform("Puppet Pin 4").position)
}

When you rig your puppet layers, your character’s left ankle is going to connect to this layer.

Now let’s set up the rest of the body parts. Here’s where that skeletal diagram comes in really handy: since you already have the basic expression format, all you have to do to set up the rest of the arm and leg controls is change the pin numbers.

Duplicate the leftAnkle expression point control by selecting it in the Effects palette and pressing control or command + D. Rename it leftKnee. If you had 1 and 4, now you need 2 and 5.

Do it all again for the leftHip, and this time change the numbers to 3 and 6.

Then, duplicate the whole leftLeg layer, and rename it rightLeg.

Press EE to show all expressions, rename them to rightAnkle, rightKnee and rightHip, and swap the numbers around for each of them. 1 and 4 becomes 4 and 1, 2 and 5 becomes 5 and 2, 3 and 6 becomes 6 and 3.

That’s both legs set up. Now it’s time for the arms. Duplicate the leftLeg layer again, and this time call it leftArm.

Rename the controls to leftWrist, leftElbow and leftShoulder, and, again, switch out the numbers. For the wrist, the pins are 7 and 10, for the elbow 8 and 11, and for the shoulders 9 and 12.

Duplicate the leftArm layer, and rename it to rightArm.

Rename the controls rightWrist, rightElbow and rightShoulder, and reverse the numbers for the opposite side of the body.

So now you have all four limbs. Time for the spine, waist, neck and head. These don’t flip left and right like the arms and legs, so it’s going to work a little differently.

Create a new null layer, set the anchor point to 0, and call this one Spine.

Create a new expression point control (Effect -> Expression Controls -> Point Control) and call this one “waist.” This one’s a little easier.

Type fromWorld( [Pickwhip pin 13’s position] )

The final expression should look like this:

 fromWorld(thisComp.layer("Source").effect("Puppet").arap.mesh("Mesh 1").deform("Puppet Pin 13").position)

All this does is grab the waist point’s location and change the numbers to World coordinates.

Next, duplicate the waist point control, rename it “neck” and change the pin number to 14.

Duplicate it again. Make this one “head” and change 14 to 15.

I like to color code the control layers, green for the left arm and leg and red for right, with a purple layer for the spine and head.

Whew. You now have a reusable puppet template! Now is a good time to save!

In the next part, we’re finally going to connect an actual puppet to this.

<iframe width=”580″ height=”325″ src=”http://www.youtube.com/embed/NZxLEXNn0ow” frameborder=”0″ allowfullscreen></iframe>

Hello, I’m Victoria Nece. I’m a documentary animator, and today I’m going to show you how to use your Kinect to animate a digital puppet like this one in After Effects.

If you have a Kinect that came with your Xbox, the first thing you’re going to need to do is buy an adapter so you can plug it into your computer’s USB port. You don’t need to get the official Microsoft one — I got a knockoff version from Amazon for six bucks and it’s working just fine.

Next you’re going to need to install a ton of different software. It’s all free and open-source, but I’m warning you now: there’s a lot of it. Check the links below to see what you need for your particular setup.

Here’s a quick overview of how it’s all going to work. Once you’re up and running, you’re going to be using a Processing app called KinectToPin, written by the very talented animator Nick Fox-Gieg. That’s where you actually capture the tracking data, as well as where you convert it to keyframe information After Effects can understand.

Then on the After Effects side of things, you’ll set up a skeletal rig for a layered 2D puppet and apply the tracking data to bring it to life.

It’s not an easy process, but the results are worth it.

Required Software:

• OpenNI + PrimeSense Sensor Module (You need both): http://www.openni.org/Downloads.aspx
• OSCeleton: https://github.com/Sensebloom/OSCeleton
• SimpleOpenNI: http://code.google.com/p/simple-openni/
• Processing: http://processing.org/
• KinectToPin: https://github.com/N1ckFG/KinectToPin
• After Effects CS3+ (CS5+ to use tutorial files)

Special Mac note: works in Snow Leopard, may not work in Lion.

After a fair amount of experimentation, it turns out it’s not that hard to do — but it does take a little tweaking. The final results are surprisingly versatile.

In this tutorial, we’re going to make the this header for a fictitious sewing blog:

GET THE FILES

To follow along, you’re going to need this brush and this layer style. (Right click and choose “Save Link As…”/”Save Target As…” to download.) You can also grab them both together as a 4kb zip file here.

START SEWING

1. Create a new file, 900×500. Fill the background with black.

2. Add the text (T) you want to make look embroidered. Use a really thin font for this — the stitch effect traces the outsides of the letters, and if you use something too bold you’re only going to get embroidered outlines. I’m working with Nobel Light. Make sure there’s enough space between your letters to avoid overlapping piles of thread.

3. Click the text warp button on the Options bar.

Choose “Arc” from the drop-down menu and set the Bend slider to about 19.

(You can skip this and the following step if you just want straight letters.)

4. Hit Ctrl+T, rotate the text a bit and hit enter.

5. With the type layer selected, choose Layer > Type > Create Work Path.

You have now created a path based on the outlines of the text.

6. Select the brush tool and load the brush file you downloaded. Change the size to 8pt or so. Make sure your foreground color is the same as your text color.

7. Create a new layer. (Shift+Ctrl+N)

8. Choose the pen tool (P), then right-click on the image and choose “Stroke Path…”.

Make sure “Brush” is selected and the “Simulate Pressure” box is checked.

Click OK. Yay, fuzzy text!

9. Load the layer style you downloaded, and apply it to the stroke layer (not the type layer).

Select the text layer underneath and make the font color a little lighter.

And there you have it. Shiny embroidered text:

10. Find a home for your fine sewing work, like this blog template:

Keep in mind that the brush preset also works as, well, a brush, and you can also use it to draw embroidered-looking things on its own — for instance, the RSS feed icon in the example. Just be sure to drop the layer style on top for shine.

Happy stitching!