KinectToAEPartII

Kinect MoCap Animation in After Effects — Part 2: Motion Capture with KinectToPin

 

This tutorial is now obsolete. Check out the new KinectToPin website for the latest version of the software and how to use it — it’s dramatically easier now.

 

Kinect MoCap Animation in After Effects

Tutorial links: Part 1: Getting Started | Part 2: Motion Capture with KinectToPin | Part 3: Building the Puppet Rigging Template | Part 4: Rigging a Digital Puppet

More info: A better way to control the puppet’s head | FAQ and Installation Guide

Project files: Click here to download the After Effects project (CS5+).

Quick note: the text is just transcripts of the videos, so you can read or watch as you prefer.

 

Welcome back. You have your USB adapter and you’ve installed all the software I linked in Part I, right? Now it’s time to get KinectToPin up and running.

You don’t have to install it, just unzip it and drop its folder inside the Processing library folder, which on a Windows machine is usually just a folder in My Documents called “Processing.” If your folder ends up called something like NickFG-KinectToPin-Bunchofrandomnumbers, you need to rename it to just plain old “KinectToPin.”

Processing sketchbook folders have to match the names of the programs, or sketches, they contain, or you won’t be able to load them.

 

Tip: SimpleOpenNI installs the same way as KinectToPin, but you need to put its folder inside Processing => libraries instead of inside just the Processing folder. You also need to rename its containing folder to SimpleOpenNI. (Thanks to YouTuber TheSquarecow for pointing out that I’d omitted that step.)


And now you’re ready to go! We’re going to run a few programs at the same time. First, plug in your Kinect, and wait for your computer to recognize it. Next open OSCeleton, then open up Processing.

Now go File -> Sketchbook -> KinectToPin and hit the play button. If you want to streamline this process a bit you can compile KinectToPin into an app, but it’s not necessary, and every now and then it doesn’t do quite what I expect and I like to see what’s going on in the code.

So right now, you’ve got one window with a bunch of text, and another with some buttons and a black background.

You can press cam to check and make sure your Kinect is seeing things. Now go stand in front of the camera. If you’ve done everything right, OSCeleton should notice you. Put both hands in the air as if you’re being arrested — that’s the calibration pose.

You should see some more text pop up on the OSCeleton side saying it’s recognized you and ready to track. There are two different ways to record the tracking data: you can hit “rec” to capture with SimpleOpenNI, or “OSC” to use OSCeleton. OSCeleton supposedly provides a smoother track, but I find it also gives me much buggier data, so I’m using SimpleOpenNI. Your results may vary. Now, press the recording button of your choice in KinectToPin, step back, and let the countdown begin. In a few seconds you should see your skeleton appear as a series of white circles.

Try to stay as far back from the camera as far as you can, and give yourself plenty of room to move. The Kinect requires a surprising amount of space. There’s a lens you can buy to shorten its focus, but it may distort your results — I haven’t tried it out yet.

The Kinect does the best job capturing you when you’re facing directly toward the camera. The open-source Kinect APIs don’t support estimating occluded joints yet, which is a technical way of saying it will get confused if you put your hands behind your back. You should also make sure to keep your arms and legs within the window at all times, or it may lose track of you.

Now, press stop to stop recording to xml. I like to make some big goofy gesture that’s easy to spot so I know where my usable track ends, since I’ve captured quite a few sequences of myself pressing stop.

Click play to watch your track play back and make sure you have what you need. I make sure to record a motion sequence several times, because I keep running into a strange bug that makes my puppet’s head fly off. And while it’s always nice to have multiple takes you really need at least one non-decapitating dataset.

So KinectToPin captures the tracking points for you in individual XML files, but you need to do one more step to get them ready for After Effects. Press the button labeled “save” to convert them all at once.

Go to My Documents -> Processing -> KinectToPin -> data -> saveae, and if all goes according to plan, your files should be waiting for you there.

Sometimes, when I have too many tracks, I need to move them elsewhere before it will let me record new ones. One way to fix this issue is to go into Processing’s preferences and increase the available memory, but if you want to delete your tracking data or move it elsewhere, you’ll need to clear out both this folder *and* the savexml one. It’s not a bad idea to move all your track data files somewhere else where you can give them proper labels.

You now have proper keyframe data, all ready for import into After Effects.

You can try copying and pasting it to a set of 15 puppet pins on a null.

If you have CS5 or higher, you can download the sample puppet from this tutorial and test your data with that.

Continue on to Part 3, where you’ll learn how to rig a puppet of your own that will react to the data you’ve just recorded.

12 thoughts on “Kinect MoCap Animation in After Effects — Part 2: Motion Capture with KinectToPin”

  1. Very cool. I don’t have a Kinect myself – but if I did I would be onto this straight away. This has massive potential… Great job getting these tutorials online and so easy to follow.

    1. Thank you! I’m really excited for the possibilities. It still blows my mind that this stuff is even possible… I remember when a depth camera was $50,000, and now I can do this in my living room with a video game accessory? Crazy.

Leave a Reply