Breathing City

Inspired by John Nelson’s breathing earth and Conveyal’s aggregate-disser post, I wondered if I could make a breathing city. Manhattan looks somewhat lung-like, so it seemed natural. Should be a fun, quick project. How naive I was.

Search and Recover
Conveyal had already gathered the data I would need to do a dot density plot, so it should be easy to find it using their post as a starting point. But wait they didn’t share links to the source files and they didn’t respond to my email. Google should solve that… hours of surfing later, I find what I’m looking for in four different places: population, employment, land use, building footprints.

Java
Excellent, now just run it all through Conveyal’s conveniently open source tool. Except its written in Java, so lets install the Java SDK. Oh and it has several library dependencies. Finding… installing… finding… (several hours later)… installing… not working. Clearly I know far too little about java to get this going.

Python Wrestling
We use python at Darkhorse, and learning some geographic libraries could be useful. Let’s use the code for the racial dot map project as a starting point for creating a python version of Conveyal’s disser tool. I just need several new libraries which translates to more hours finding, installing, re-installing, searching, uninstalling, removing, copying, installing again because computers are petty and vindictive. Then finally shapely and osgeo are working… yay!

Baby Steps
Now we just take several hours to learn how not to use these libraries and eventually we stumble across one or two things that work, then a couple more, and then we crawl toward a messy program that does what someone else has already done, but at least I speak this one’s language.

Combine CSVs with Shapes
What? Excel stopped allowing you to edit and save DBF files, when did that happen? This tool is a bit buggy, but it brings that feature back.

QGIS forces antialiasing
You can’t turn it off. If you want to create single pixel markers, it just won’t let you color them properly, I tried for far too long (if you know how, please let me know). Good thing Excel is a poor man’s GIS (Yep every one of the frames was made in Excel)

Find More Data
So now I can make a plot that doesn’t breathe. But I want to show change in the typical workday. I’m gonna need more data for that. Several searches and false starts later we find work related activity percentages by time of day. Manhattan probably has a different profile than the US average, but close enough.

Find even more data
But each dot is a person, so I can’t just have them flicking on and off randomly to match the time of day percentages. They need to go to work for a while then come home for a while, so I need to give each dot a schedule. Maybe this is overkill, but I can’t stop now. More searching finds us a rough hours of work distribution.

Solver time
Now I’m forced to assign schedules to the ~1.5 million people living in Manhattan and the ~2 million people working in Manhattan. But the sum of those schedules needs to resemble to hours of work distribution and the percentage at work for each hour of the day. Time to break out Excel’s solver engine. With it we can create ~200 schedules with probabilities to match those profiles. Then we can distribute them to each of of our people.

Data is done
Finally we have data for each of 24 hours for both home and work. We’re making some huge simplifying assumptions (e.g. Manhattan’s work profile is the same as the rest of the US, when people aren’t at work they are at home, there are only 200 possible ways to spend your day, when we build this someone will want to see it) but we have a reasonable data set.

Design
Now we just make some maps and push pixels around on the screen until they look good, then painstakingly create 24 versions to string together for the animation. Eric Fisher has some great tips for making and coloring dot density plots in this post. Then we’ll add a bar chart, no an area chart, no a line chart, wait… I’ve got it, a mesmerizing heart rate monitor looking thingy to go with our breathing theme. Nice.

Relief
See, it is super easy and takes almost no time at all to create something like this, as long as your definitions of “super easy” and “no time” are flexible enough to include difficult and time-consuming.

This entry was posted in Analytics, Visualization. Bookmark the permalink.

21 Responses to Breathing City

  1. Wow, very nice! That sounds like an aweful lot of work. I wonder if R with the packages ggmap and animation would be good at creating something like this…

    I really like your conclusion – it made me laugh ;-)

  2. John Nelson says:

    Great work, Joey; this is beautiful!

  3. Nice work! I made something similar a while ago by animating business hours as heatmaps. See it here: http://virostatiq.com/animated-business-activity-slovenian-cities/

  4. l.jegou says:

    Very interesting, but i think it lacks a proper legend.

  5. Brandon says:

    Howdy Joey. It’s Brandon from Conveyal; I wrote aggregate-disser. I’m really sorry to hear that you had trouble getting in touch. Our blog could use a comment section. If you have any questions I’d love to answer them – badhill@gmail.com

    -B

    • Joey says:

      Hi Brandon, thanks for reaching out. No worries, I was just trying to expedite the data collection process, I found it eventually. I also realized after posting that you wrote this http://bmander.com/dotmap/methods.html which lead me to the github code I used as a starting point for my python version of your aggregate-disser tool. Nice work on all your dot maps, as well as the impressive portfolio of projects, and thanks for taking the time to write about them and share the code.

  6. Mathieu Pellerin says:

    Regarding QGIS’ absence of a [x] switch off antialiasing option, there’s an open issue filed here: http://hub.qgis.org/issues/9281

    It would probably be useful (and motivating towards the devs) for you to add a comment to explain your use(s) for such an option.

    Beyond that, what a great map this is! :) Have you ever looked at QGIS’ TimeManager plugin? It’s pretty powerful.

    • Joey says:

      Mathieu, thanks. I had’t looked at TimeManager, thanks for pointing it out. I’ll also have a look at that issue you mention.

  7. Ima Jerk says:

    Very cool. But what about the people who work in Central Park? ;)

  8. Beautiful, and how appropriate that the heart of this breathing “lung” is Central Park’s land, trees and water! I’d love to see the year-round and daily activity patters there. Next project?

  9. RobertS says:

    I find the way the Columbia University campus lights up during the day a little weird, unless it is a case of students “going to work”.

    Some of the other uptown “spikes” make perfect sense, though, as they are medical complexes.

  10. Tom says:

    Wow, beautiful data visualization! How hard would it be to extend this to the entire city with reasonable definitions of “super easy” and “no time”?

    • Joey says:

      Tom, It wouldn’t be hard to create the data set necessary for the whole of the city, however, I don’t know that excel would be able to handle the extra size. Another program would need to be used to create the higher res images.

  11. Nate Wessel says:

    Obligatory cartographer question: So, this is unprojected lat-lon values transposed directly to excel’s coordinate space? Or did your data come pre-projected in something like a New York State Plane coordinate system?

    Why did you need un-anti-aliased rendering in QGIS?

    I can’t believe you did this in excel…you’re a total masochist! Have you looked at R?

    One more suggestion: I suspect your color ranged is clipped at some point(s). This data on a linear scale would presumable result in a white hot point at something like the empire state building or times square and the rest would look black. A legend would help, or a log scale perhaps.

    Critical questions aside, this is neat stuff! I’d love to see what you’d do with a real GIS :-)

    • Joey says:

      Nate, Yes this is unprojected lat-lon values put directly into excel so it is somewhat like an equirectangular projection I think.
      The anti-aliasing in QGIS prevents creating fully saturated markers that are 1 pixel by 1 pixel. For some reason that 1px gets filled not with the color you have specified but with the ‘anti-aliasing color’ that the border color would contain if it were larger than 1 pixel.
      We have some people who use R in the office, but I’m not one of them… yet. I hope to spend some time educating myself in the near future. Excel was handy and, given my intimate knowledge of it, not that difficult once I’d given up on QGIS.
      Yes the color range is clipped, though not as much as you might expect. First we are not using a linear scale, its actually a squared scale, as suggested by Eric Fisher in the article I linked to. This allows quite a range of values to be visible. Any pixel that has more than 100 people in it is fully saturated.
      I had thought about adding a legend, but decided against it. Because of the many assumptions made in creating the data set, I didn’t want to give the impression of accuracy in counts that just isn’t real. The focus is more on the patterns than the values.

      • Nate Wessel says:

        I bet you could actually project it fairly easily, perhaps on a mercator or something else conformal and take Excel to a crazy new place, by applying some relatively simple math before creating your pixel bins. You’d just need to transform the longitude values–I think. That would eliminate the slightly squashed effect, but you maybe wouldn’t want to try it on a small scale map.

        I’m with you on the legend, though I think knowing it was a root scale would have been helpful. City densities are nothing if not jagged and full of outliers, and this map looks surprisingly smooth.

        Keep up the good work! And learn R, damnit. <a href="http://cincymap.org/blog/gif-mapping-differential-passenger-flows-preliminary-results/"I just started making my first map gifs with it and it was sooo quick to loop through and write 700+ frames.