maybedonemap

An Open Transit Map for Hudson County

Is the PATH down again and you need to find another way home? Want to show how limited transit options are in your neighborhood so you can fight for better service?

I’ve been hard at work on something that might help: a reasonably complete open map of all mass transit in Hudson County, free to use and adapt for your own purposes. At the moment the data’s set up as a series of shapefile layers in QGIS, and the project’s just been updated to work with the latest version (hyperthreading = so much faster).

new-new-header-image

When I say “reasonably complete” I mean it — trains (NJT + PATH), light rail, buses, ferries (NY Waterway, Liberty Landing, Statue Cruises),  even jitneys and smaller private bus companies like A&C.

JSQ

There is more work to do, however: at the moment the map is still missing the Hoboken HOP bus, NY Waterway’s shuttle buses, and one bus from Pennsylvania that stops twice a day each way at Exchange Place. It’s also possible I’ll discover some other secret buses, as that’s happened several times so far — Hudson County is full of poorly documented transportation routes largely run on word of mouth. (You know it’s bad when I’m actually excited to discover a company’s website contains maps drawn in MS Paint, because at least that means there ARE maps).

The big goal of this project is to tie everything together in a clean digital format that can be adapted for a wide variety of uses. (That includes commercial use: if the smaller bus companies want to generate their own maps for riders based on the routes I drew for them, THAT WOULD BE AWESOME.) Eventually the plan is to build some sort of slippy map you can interact with on the web, but right now we’re still on the “get everything in one place” stage.

This project started as part of OpenJC‘s Hack for Change National Day of Civic Hacking (I won!), but I’ve continued working on it for several weeks now. And I have learned A LOT about QGIS in the process. The best single tip I can offer is to use the OpenStreetMap plugin to download a dataset, but to skip the spatialite db conversion steps and just import the .osm as a vector layer. It seems a bit slower on the redraw, but no more missing streets!

I’ve also learned a lot about hierarchical design for complex maps. When I’m designing maps for television, they need to be simple and easy to interpret at a distance, so I’ve never had to fit so many different types of things on screen at once. Getting them all to harmonize in any remotely readable fashion has been a real challenge, but I think I’m getting there. I expect in the end many users will not even want all the layers enabled at once, but I’m trying to make things slightly less spaghetti-like if they do.

Also, if you randomize a bunch of settings, you can generate some neat abstract art with this thing:

random-color-map

abstractbuses

Nighthawks Piip-Show

Dramatis Birdsonae: The Birds of Piip-Show

I’ve been working late a lot, and the friendly birds of NRK’s Piip-Show have been keeping me company. But the species guide is in Norwegian (obviously, it’s a Norwegian show), and the usual translation tools do a terrible job with it.

So I made a little Tumblr to help curious English speakers identify the birds. I hope someone finds it useful, or at least entertaining. The disappointed woodpecker is my current favorite.

Airmail Route Map

Mapping Plugin Roundup

Can’t be bothered with learning GIS software? Here’s a quick roundup of a few plugin options when it comes to map creation for animation. Have I missed any others? Let me know and I’ll add them to the list.

After Effects

Fabiantheblind’s AEmap and Locations plugins give you the ability to build, customize and animate a political equirectangular map, and/or  drop geolocated coordinates onto several different projections. These are great if you have country-level data to display or want to pinpoint specific locations. (He’s working on a tool for InDesign too!)

Cinema 4D

Paul Everett’s cinemaplugins.com sells several tools for working with map data directly within C4D. His plugins can import both DEM files (digital elevation maps) and OpenStreetMap data automatically.

Illustrator + Photoshop

Avenza MapPublisher is a full GIS toolkit for Illustrator, and  Geographic Imager is its raster companion for Photoshop. These are serious tools for mapping professionals, and priced to match. There is a more limited “basic license” available for Geographic Imager though that’s only $199.

Blender

ESRI Shapefile Import/Export and Georaster Import are free(!) plugins for Blender that do exactly what it says on the tin: they let you import shapefiles and geolocated raster images into Blender.

Pacific - Sample

Adventures in map animation with QGIS, After Effects/C4D and Illustrator

Before I start this tutorial I need to complain about something:
The Pacific Theatre of WWII is a huge pain to map. We were fighting over some SERIOUSLY TINY ISLANDS. Even just finding a vector map that includes them is difficult, and when you go to add labels, well… zoom out wide enough to see where in the world you are, and you can’t see the places you’re trying to pinpoint.

QGIS to the rescue! Yes, it’s finally time to make our own maps.

I’m not gonna get too technical here (I’m still learning this stuff myself), but this will hopefully serve as a useful workflow reference to help animators on their way to becoming DIY cartographers.

So what is GIS, anyway?

You’ve heard of data-driven design? This is some of the stuff that’s driving the data for data-driven design. GIS stands for Geographic Information Systems — basically, all things digital mapping. If you’ve only ever worked with maps designed by someone else before, learning to use GIS software will enable you to go much deeper: not only can you create your own maps from raw data, you can measure exact distances, do all kinds of cool analytical work, go nuts with custom styling and even change map projections on the fly.

There are lots of different GIS applications, but we’ll be using QGIS because it’s free, open-source, and outputs files in a usable format. You can download it here.

Grab some data

To get started, you’ll need to import some map data. Shapefiles (.shp, usually accompanied by additional files with other extensions) are the main way to get outlines into GIS software. These are basically just geographically-linked vectors, and there are tons of them available for you to use.

Here are a couple sources for shapefiles you can use for anything you want:

You can also use OpenStreetMap data freely. If you’re working with a small enough area (think neighborhood, not country), you can even grab the vectors you need right within QGIS. Here’s a nice tutorial using OSM to generate a map of London’s pubs that covers that import process.

Another way to get data into QGIS is to import KML files. This is the way to work with points you’ve mapped using Google Maps Engine or Google Earth. Maps Engine is a lifesaver if someone gives you a giant spreadsheet of poorly labeled locations. You can upload a CSV (100 locations at a time with the free version, but breaking a big dataset into chunks is easy enough), tell it which columns contain the location info, and be amazed when it figures out what the guy who sent you the spreadsheet meant when he abbreviated California as “Calf”. mapsengine

There are also thousands of maps made by other people whose points you can borrow. Once you get everything correctly positioned on the map, click the folder icon in the top left corner and choose “Export to KML”.
kml-export

Import your data

Now that you have some data to work with, open QGIS and create a new project.

Add your dataset by choosing the appropriate import button for its file format. The importers are located on left side of the screen as well as in the Layer menu, where they are more helpfully labeled. addvectorlayer

Add Vector Layer is likely to be your go-to choice — that’s the right option for shapefiles, KML and CSV data, among others.

If you attempt to import KML from Google Maps Engine and get a blank layer instead, there’s a workaround: open the file you downloaded in Google Earth, resave as KML and try again. (Note: Google Earth may give you a KMZ file, which is a zipped KML. Just change the extension to .zip and unzip to import into QGIS.)

The earth isn’t a perfect sphere, so you’ll need to choose a geographic coordinate reference system (CRS) to make sure your points land in the right place. WGS84 is the standard you’re likely to be working with.

As long as it knows its original format, QGIS can reproject data on the fly. You can set these options in the CRS settings in Project > Project Properties. QGIS ships with a couple thousand options, and if those aren’t enough you can make your own custom projections.

CRS

You can also add points directly within QGIS through a process called geocoding. Enable the Geocoding plugin by going to Plugins > Manage and Install Plugins, and you’ll be able to search for locations one at a time and pin them to your map. (If you have a ton of addresses to batch through, you can follow this tutorial instead.)

geocode

geocode-japan  geocode-tokyo

You can also “reverse geocode” — click on a point and ask QGIS to label it. I’ve had a handful of issues with this plugin, namely that my added points disappear after a program restart, so you’ll probably want to export your Geocoding layer as a shapefile then reimport it. It’s also still using the deprecated labeling engine, so things may scale weirdly — you can switch to the new labeler in the layer settings.

There are a wide variety of other plugins that can be installed the same way as Geocoding. I’ve found a couple especially handy: QGIS doesn’t really do curved paths, but I needed to create a bunch of arcs connecting points to a single city — think airline destination map style — so to get that working, I:

  1. Exported my locations via MMQGIS‘s Geometry Export to CSV menu, manually added the target city to each with the same ID valuelocations-for-points2one
  2. Reimported and ran that file through Points2One to generate the connections as straight lines.straight-lines-b-29
  3. Once I output the lines, I used Illustrator’s Warp:Arch effect to make them into curves, then animated them in After Effects. Convoluted, but it worked nicely.Map With Curves - B-29 Contractors

 

This “do a thing – export the layer – reimport as a new layer – delete original” process probably sounds weird to you, but it seems to be the primary QGIS workflow and we’ll be using it a lot. Advance warning: you’re going to end up with a lot of fiddly little shapefile files. You should probably decide where you’re going to keep them now, before they end up scattered across your hard drive.

Position and style your map

Moving around in QGIS isn’t that different from, say, Google Maps — with one big exception. Global shapefiles are usually centered on the Atlantic, and if you need to generate something Pacific-centered you can’t just roll the map around to the other side. I followed these instructions, which are tricky but seem to work. (If you know of a better method, please tell me!)

pacific-centered-map

To customize the look of your layers and/or add labels to your points, double-click on the layer name. You’ll find a million options inside. QGIS even supports blend modes, which look great but don’t work with vector export — they’re still super-helpful though for previewing purposes. To change both the line and fill styling, click on Style, then click the default Simple Fill in the Symbol Layers list to get to the full parameter set. From this screen, you can also access more advanced options. Make sure to check out the line and pattern fills; they do some neat things if you’re going for a more abstract look for your map. Here’s Australia with polka dots and kittens:
australia-polkadots

australia-kittens

Get your vectors out of QGIS

And now for an annoying extra step. Once you have a map you’re happy with, you can’t just go “Save as > PDF”. Instead we’re going to be using a tool called the Print Composer for this bit.

Note: If you’re attempting to create a map of the whole world, you should avoid using the highest-resolution land data from Natural Earth. I can’t get it to import properly into Illustrator for the life of me. I can, however, use the high-res shapefiles as long as I’m only exporting a smaller portion of the globe. They’re really nice to work with when you’re dealing with all the aforementioned tiny Pacific islands.

Go to Project > New Print Composer and give your new preset a name.

Click the Add New Map icon (looks like an unrolling sheet of paper), and drag a rectangle the size of the page. If you want to match your current view in QGIS, and it looks like the map’s the wrong size, go to Item Properties and click “Set to map canvas extent.”

mapcomposer

Now you can save as a PDF.

If you plan to map your data onto a sphere (either in C4D or with CC Sphere), export your file as an equirectangular projection, with the extents set to X min = -180, Y min = -90, X max = 180, Y max =90. You’ll probably need to crop the resulting file/change the artboard size, too, since QGIS exports documents in the shape of the paper it’s “printing” them on. 3600×1800 is a good choice of final image size, since it’s easy to keep the math consistent if you need to add something later by hand — just multiply your degrees by 10.

Open your map in Illustrator

Here’s where we’re going to turn a flat vector map into something we can use in After Effects.

illustrator-openmap

When you bring the PDF into Illustrator, everything comes in nested within a single layer. You’ll have a lot more design flexibility later on if you take a couple minutes to break up the map components into multiple layers. When there are lots and lots of paths, I find the easiest way to pull the pieces apart is to select one part of one element,  go to Select > Same > [Relevant appearance attribute], then cut and paste in place (Ctrl + Alt + V) on a new layer. If you want to be able to link to your points in AE, use the Release to Layers feature, then drag them out to be top-level layers.

illustrator-selectsame

illustrator-layers

You can add additional styling at this point, too. Using the art brushes on country border strokes is an especially nice effect:

illustrator-restyle

Label your layers, and save as an Illustrator file, or export your image in your favorite C4D texture format.

Protip: To import Illustrator paths straight into C4D, you need to save them as Illustrator 8.

Now on to After Effects and/or C4D

This is the easy bit. Well, easy for you, assuming you already know After Effects or C4D. For AE, just import your AI file as a composition, drop the layers in your comp and animate and composite away as you would any other map. Here are stills from two relatively simple variations on the same dataset:

Pacific - Sample

Pacific - Sample2

Map - Saipan A

Map - Saipan B

The only real differences between the two are color and texture — you can go much further than this.

Even though QGIS’s text labels aren’t much good for animating directly, I tend to import them anyway (I usually stick them all on one layer). They’re a handy a guide for creating native AE text labels. You can do the same thing with points, or import them as individual layers so they’re easy to alt-replace with precomped animation like the concentric rings above. Having georeferenced guides to refer to even when you’re creating new elements from scratch opens up a lot of possibilities. If someone makes a late change to graphic (never happens, right?), it also makes map additions so much easier: you can go back to QGIS, use your saved Map Composer preset to export new borders or geolocated points, and they should line up exactly.

If you want a little bit of 3D and have either 1) a good enough GPU or 2) enough patience to use the Ray Tracer, you can choose “Create Shapes from Vector Layer”, switch to RT rendering and extrude the shapes. It takes forever to render, but you can do some interesting and/or very weird stuff this way. This also lets you apply Shape Layer effects like Wiggle Paths.

For C4D, you have a couple of options:

Image as texture (those of you with just C4D Lite: you can totally do this in Lite): create a new material, import an image file of your map as a texture, throw it on a sphere, style as desired. Add some splines connecting points and you have a snazzy 3D globe. Alternatively, import the AI paths and do a whole bunch of extruding for a fancy 3D ‘flat’ map that renders faster than with the AE Ray Tracer.

Wrap your vectors around a sphere (you’ll need more than Lite, as this uses the Wrap deformer): follow the instructions here.

Things to watch out for

  • When it comes to the shapefiles inside your project, QGIS is a DESTRUCTIVE EDITOR. I learned this the hard way: I accidentally deleted the whole world at one point. If you want to move some borders around, right click on the layer and Save As… before you go into edit mode.
  • The QGIS team is in the process of replacing their labeling engine, and as of 2.2.0 the new one automatically converts text to curves on export. If you want to preserve the labels as editable text, you should use the deprecated label system — or wait a few weeks. They’re supposed to be fixing this.
  • Don’t get too fancy designing backgrounds in QGIS. They only export for me about a third of the time. Do that stuff in AI or AE — or Photoshop.
vegasaerial

NAB 2014: What I learned

Just got back from this year’s National Association of Broadcasters conference in Las Vegas. A few thoughts:

The After Effects and C4D communities are full of incredible and generous people.

Next year, everyone needs their Twitter icons on their badges.

I’m finally allowed to tell you I made the Goatmeal Stout splash screen. I am way too proud of this. I made a tiny hidden piece of After Effects, guys!

It is possible to attend a training session solely for the purpose of trolling someone who couldn’t go.

BitTorrent Sync is faster than FedEx for distributed production workflows.

Some of my crazy ideas are actually possible.

My favorite new product is one I’m not going to use personally — it’s one that provides me a one-word answer to a question I’m asked all the time. Musicians who want to film performances: buy a Sony HDR-MV1. Done.

It is possible to be so tired you can’t sleep.

I will sit through a demo of anything if you give me a free cup of decent coffee.

The Oculus Rift still makes me ill. But that out-of-body experience was worth the dizziness. Apparently they “fixed” the booth setup not long after I tried it, but looking around and seeing yourself standing a couple feet away, then looking down and seeing you don’t have legs? Absolutely wild.

It’s not NAB until you’ve snuck into something. Hi, I’m “Karen Flowers”.

Twelve years after it happened, I finally have photographic proof of my best crazy job story (Thanks Eric!). Here’s why I got laid off from an editing gig in FL:

Don't worry, the guy who fell on my desk was ok. But that was the end of my gig.

Not my fault. Electrician fell through the ceiling onto my edit bay. He was ok, computer not so much.

The “WWII from Space” maps are much nicer than mine, but they couldn’t find the shapefile I’m looking for either. This is oddly reassuring.

Go-karts cause shoulder welts?

More ladies, please. I think I met nearly every female motion graphics person at the show, and there aren’t nearly enough of us.

What happens in Vegas during NAB probably gets filmed in 4K.

A lot of people I didn’t realize listen to Actually Happening listen to Actually Happening.

NAB is faster than Google for solving your technical problems.

The City of No Clocks is a poor choice for getting over jetlag.

You can pack light, as long as you accept all free t-shirts offered.

The giant frog is singing “Low Rider” now. But the Lake of Dreams closes early.

A non-smoking room at the Riviera is just “whether or not the person before you smoked in it”. Don’t stay there unless you’re checking in early enough in the day to get a renovated room. They don’t hold them even if you have a confirmed reservation. Better yet, don’t stay there.

Networking isn’t hard when it’s with people you actually like.

Memory foam shoes are magic but still not enough to save you after three days walking the show floor. Ow.

Hanging out with people nearly half or twice your age is fascinating. Be the elder statesman giving advice to college kids trying to figure out their careers, then feel like a total newbie listening to tales of the old days. Someone met a guy who has been to 47 of these things.

It is possible to keep Kevin occupied for days on end with just a piano and exceptionally good tacos.

Trust the recommendations of locals. Tacos El Gordo, between the Wynn and Riviera? See above.

Honest bus announcements: “Next stop, more places to drink and gamble, basically.”

How did anyone do this before Twitter?

Thank you to everyone, friends new and old, for a wonderful time. See you next year!

Documentary Animation | Motion Graphics | Design