Kinect MoCap Animation in After Effects
Project files: Click here to download the After Effects project (CS5+).
Quick note: the text is just transcripts of the videos, so you can read or watch as you prefer.
Yes, it’s the part you’ve all been waiting for: it’s finally time to set up your character layers!
This was originally going to be in Part 3, but there are so many steps I realized I needed to break things down a bit more.
About the Puppet Tool
We’re going to be using the Puppet Tool a little differently than it was originally intended to be used. It was designed for a single layer puppet, but when you try to animate a character that way it gives you a very wiggly result — what I refer to as the “Bendy Elbows” problem. It’s great for deforming or reshaping a layer, or making old-timey cartoons, but it won’t give you anything close to realistic movement.
And if you try using all 15 points of your KinectToPin tracking data with a flat puppet the result is.. unfortunate.
However, if you only have two puppet pins on a layer, you get a completely different effect: instead of making it go all wibbly, they work sort of like multiple anchor points with the layer stretched between them.
So that’s why we’re putting the source data on one layer and using expressions to link to it to control separate pieces.
Designing Your Character for Kinect Control
As if this process wasn’t complicated enough already, it also makes building your character a little more work: instead of just creating a single layer, you need to build your puppet out of lots of little pieces so that it can rotate wherever there’s a joint — you need a hand, a forearm, an upper arm and so forth.
One thing is a bit easier though. Instead of having to use the Puppet Tool’s layer overlap feature, you can change what part of your character is in front by simply changing the layer order. This also means that if you have an element that needs to be in front of some layers and behind others, like the head of a character with long hair, you need to make it two separate layers.
You can use your own preferred technique to create your puppet, whether it’s drawn or cut up in Photoshop, created in Illustrator or even generated via After Effects shape layers like this one. I’ve used this process with everything from Illustrator vectors to pieces cut out of vintage photos and artwork, but for this demo I used shape layers so I could give out a single .aep file with no dependencies. You can’t just cut up a flat puppet or photo, unfortunately. You need to give each layer a little bit of overlap with the ones it connects to so the puppet doesn’t come apart at the joints.
Also, if you’re importing your layers from another program, make sure to crop them to the layer dimensions on import (not the comp’s) so you aren’t dealing with unwieldy giant rectangles the whole time.
Scaling: Getting it Right Before You Rig
Regardless of your design technique, be sure the layers are decently high resolution so you have some flexibility in how close you can zoom in on them. Even if your character is made of vectors, you’re going to lose some sizing flexibility when you add the Puppet pins. Here’s why: when you first add pins, the Puppet Tool works by creating a mesh of your layer at that layer’s original resolution. And you can’t use Collapse Transformations on a layer with a Puppet Tool instance — it breaks everything — so you want each piece as big as possible at this point. All the scaling down needs to happen inside the puppet animation comp, and all the scaling up needs to happen once that comp is nested elsewhere. Don’t worry if the raw elements are actually bigger than your animation comp itself: even though the puppet comp is only 640×480, you can scale back up into HD later as long as you plan things right at the start, because you can apply Collapse Transformations to the puppet comp once it’s nested in a larger comp.
In short: make sure all your layers are nice and big at the point the Puppet Tool gets to them.
If you’re using vector or shape layers, you need to precompose them before you apply the puppet tool. Same goes for if you have a layer that’s going to contain animation inside itself, such as lip sync on a head. Make your precomp the size of the layer at 100% scale, not the size of the puppet comp, or you’ll lose the ability to enlarge it later.
Setting Up After Effects for Rigging
Okay, so, you’ve got everything loaded in, precomposed, and the right scale? Good. Now you can connect it to the template you built in Part 3.
You’re going to be linking Puppet pins to the expression point controls in the template. The easiest way to do this is to open up multiple Effects palettes for the different control layers down the side of the screen. You can do this by clicking the lock icon in the Effects window, then opening the next one below it, locking that one, and so on. Having all the points visible makes it easy to pickwhip the right one.
Connecting to Control Layers
Let’s start with the left lower leg. I tend to hide all the layers I’m not currently rigging so I don’t get confused. Use the Puppet Tool to create a pin where you want the ankle pivot to be, then one where you want the knee pivot to be.
When you created these pins, the Puppet Tool also created the mesh it’s going to use to deform the layer, and you can see it by clicking the “Show Mesh” checkbox. If you want to make sure the edges of your limb don’t get cropped off, you can set the mesh expansion up to five or so. If things seem really slow, you can drop the number of triangles way down and that should help, but if you have too few triangles things won’t curve smoothly.
Now, hit U to show the keyframes on the timeline. For Pin 1, alt-click the stopwatch, then type fromComp( [pickwhip leftAnkle control point from leftLeg layer] ). The “fromComp” transform makes the Puppet Tool understand where to put the pins relative to the layer’s dimensions instead of to the comp as a whole.
The expression should look like this:
Now, do the same thing for Pin 2, but link it to the left knee:
Now, scale down the layer until it appears to be a sensible size and proportion.
If you built your character all at once, you’re going to scale down all the layers to this size. Try scrubbing through the timeline. Congratulations, you’ve successfully animated a shin!
You’re going to follow the same basic pattern for the upper and lower arm and leg pieces: add a fromComp layer space transform, then choose the relevant point control. The torso, hands, feet and head work differently, though, so save them for last.
So: left upper leg. Point on the knee, point on the hip joint.
The knee pin gets the same expression it had on the lower leg layer. The hip joint gets:
Hey check it out, you have a leg!
Same deal on the right side of the body. Add the right ankle:
And the knee point:
Now add the knee point to the upper leg:
And the hip point:
Starting to make sense now?
Now we’ll do the arms. For the forearm, add pins to the right wrist:
And the right elbow:
And for the upper arm, you need the right elbow:
And the right shoulder:
That’s the right arm done. Now for the left.
Left forearm: left wrist:
And left elbow:
Left upper arm is the left elbow:
And the left shoulder:
Scrub through: it’s starting to come to life!
Next we’re going to rig the torso, which is a little more complicated.
We’re actually okay with it bending and flexing, so it gets five points: the hip and shoulder pins at the corners, as well as the waist pin right in the middle. Connect them the same way you linked up the limbs. Here are the five separate expressions:
Scale that down, too:
You could also add the neck pin if you wanted, but whether that works well really depends on the design of your character.
Adjusting Pin Placement
Now, take a look at how things are moving. If it seems like a shoulder pin is in the wrong place, you can delete it and create a new pin in a different spot. The Puppet Tool will keep using the same underlying mesh. Copy the expression before you delete the pin so you can add it to the new one right away. One thing that’s really important to point out here is that if you make any changes to the shape of the layer itself, you need to delete more than just the puppet pins — you need to delete the entire puppet effect instance. Otherwise, it’s going to keep on using the old, outdated layer mesh. This is also why you can’t copy and paste my puppet pins to your own character.
The Hands and Feet
Next we’re going to attach the hands and feet. These are quick and easy: you don’t need the puppet tool for these, and you don’t need the fromComp layer space transform. However, instead of a pin, you’re going to apply it to the layer’s position transform. Just pickwhip the right arm control, like so:
Now, move the anchor point until the hand is connected to the lower arm in the right spot. The hands and feet are the only things you’ll actually need to animate by hand to match the Kinect-captured movements. Just set rotation keyframes so they’re in the correct position. If you download my sample puppet, you may notice I have Time Remapping enabled on the hands as well. Stay tuned for Part 5 to learn how to use that to change the hand pose.
Now, do the same on the left side:
And now the feet. Right:
Rigging the Head: Two Options
The last thing we’re going to rig is the head. You have a couple of choices at this point. If you want to animate head tilt yourself, or have a head that’s going to change shape from talking or varying facial expressions, you should attach it the same way you did the arm and leg by pushing the head’s anchor point to the bottom and linking its position transform to the neck point control:
Scale to match:
If you want to use the puppet tool instead, add one point at the neck and one in the center of the face.
I find the head data is the most unreliably tracked, so I’ve split the x and y positions up so I can offset them individually if need be. In this case, here’s the expression I used for the neck in order to move the Y point 21 pixels:
x = thisComp.layer("Spine").effect("Neck")("Point"); y = thisComp.layer("Spine").effect("Neck")("Point")-21; fromComp([x,y])
Don’t forget the fromComp transform at the end!
And here’s the one I used on the head point:
x = thisComp.layer("Spine").effect("Head")("Point")+2; y = thisComp.layer("Spine").effect("Head")("Point")-3; fromComp([x,y])
There’s no magic to the offset numbers I used… they’re just the numbers I found worked through trial and error when applied to this one particular track. Stretching a face even a pixel or two is noticeable in a way stretching a forearm isn’t, so you just kind of have to eyeball it.
You’re all set! Hit play and check out the results, or to see it nice and big drop it into an HD comp and enable Collapse Transformations, like so:
You now have a fully rigged Kinect-controlled puppet, and all you have to do to change its actions is to paste new tracking data to the source layer.
After a little more tweaking, here’s the final result:
Stay tuned for Part 5 for finishing techniques, including a more advanced way to animate the hands and feet, a quick and dirty trick to automate lip sync, and how to bring your puppet into HD.