Thursday, March 26, 2009

Converting a shapefile into one kml file per feature

Below is some python/GDAL code for an easy way to split a GIS shapefile (such as those found in the CA Statewide Database) into a bunch of kml files, one for each feature found in the file. For example, this can be used to create individual precinct maps from the county and whole state files in the SWDB, which can then be rendered in either Google Earth or Google Maps.


This code is a long way from perfect; it's finding some MultiPolygons where there should be polygons, and GDAL barfs out some missing symbol errors (but keeps on truckin' anway). But I didn't see a ready solution for this posted elsewhere, so enjoy (sharp sytax highlighting via http://pygments.org/):



There are a bunch of different ways to get into the guts of a shapefile and start mucking around with it. If you have an ESRI ArcMap license you can sort through the numerous ESRI .Net APIs, which is how I've ordinarily done it. But it can be slow going, and you do need the license. As far as I can tell this has to be the easiest, fastest way to get there with open source. And writing python is a lot more fun C# or VB anyway.


If you'd like to try this at home, this GDAL API tutorial from the geodjango site is invaluable. And if you're on a Mac, for the love of all that is holy don't try to compile GDAL and all its dependencies. Download the "KyngChaos Binaries" instead. Yes, they sound like warez but they seem to work fine and will save you hours of wandering through Makefiles.

Monday, March 2, 2009

"It's gonna fail."

"People ask me if I hope [Obama's economic plan] is gonna fail. I tell 'em, I don't have to 'hope' anything. It's gonna fail."-- Tom DeLay

One week from today, I am going to place either a formal prediction a bet over at Long Bets, a site designed to keep track of exactly this sort of thing. Here's the draft wording of it:

Prediction: The Obama administration will exceed the Bush administration's performance over all of the following social and macroeconomic indicators: better rate of job creation, better rate of GDP growth, lower poverty rate, lower child poverty rate, greater rate of S&P 500 growth, fewer total number uninsured, fewer net oil imports, less public debt, less public debt as a percentage of GDP, and greater per capita attendance at four-year college.

Tom Delay is only the tip of the veritable iceberg. Many conservatives are saying that the President's agenda WILL fail, that they hope it fails, and that people should even sign up to work for it to fail. Although Rush isn't even saying whether it would be a good or bad thing for it to fail, which makes me think that perhaps the words success and failure are just getting beaten to death here.

So ok: time to put up, folks. I'm making it easy for conservatives and tough for Barack. He has to do better than Bush on all ten metrics. And in a few cases, Bush put up some solid numbers, like a 41% increase in GDP. If Barack is really destroying "economic freedom" and wealth on a heretofore unthinkable scale, he can't possibly do better than that, right? This is hardly fair. It should be an easy bet.

I've tried to make these numbers conservative-friendly. Other than jobs, health care, energy independence and education, these aren't particularly my priorities. For example, one of my priorities is income inequality, which is usually measured by something called the Gini index. But since a solid source of year over year US data for it isn't even available on the web, I'm not including it. I could personally almost care less what happens with the stock market, although the amount of pain that people who are near retirement are experiencing is unfortunate. (Even if I did have money, I wouldn't be in the market for the same reason I don't gamble much when I go to Vegas. Both Vegas and Wall Street can be a lot of fun but that they mainly exist to eat small players for breakfast.)

But the fact is that I think the Obama administration will do right by Wall St. once the current mess is fully unwound - he's just not going to do it at the expense of Main St. Like Barack said in his acceptance speech, "for those of you whose support I have not yet earned, I intend to be your President too." So I'm including the S&P 500 as well as the GDP. Flawed as the GDP is as a gauge of what matters, reliable, recent GPI (genuine progress indicator) figures also aren't available, so I'm not including that either. Nor am I including total hours worked per week, which I'd like to see decrease. And I'm staying away from social issues, although I'd guess we'll have more guns and fewer abortions than the Bush years, too.

I was thinking of including some numbers regarding how much better the Obama administration will do, but it's taken long enough to pull this together already. And with Obama starting down by three touchdowns because of the mess that more than forty years of conservative economic orthodoxy has made, he's got his work cut out for him. If someone wants to take the bet but only on conditions of proposing how much better Barack will do, fine, propose some numbers and I'll think about it. And if someone wants to crunch similar numbers for the Reagan adminstration, I'd consider that bet too.

There is a very real chance that the bottom is going to completely fall out of things. A $50T+ market is collapsing and we've been underinvesting in society for decades. That is putting us in uncharted economic territory. It'd probably be smart to wait a couple of weeks and see where things go. But I'm not being smart. I'm making this bet now, when things are looking all bad.

The details are below. If anyone wants to dispute these numbers or add a few more before taking the bet, please leave them in the comments. (and please feel free to check my math!) I tried to use direct government figures wherever available, and as close to January 2001 to January 2009 figures as possible. If you'd be willing to take the bet, propose an amount. I propose that the money go to build.org, which seems like something both progressives and conservatives can agree is a good thing.

Incidentally, if you're working on government transparency, the difficulty of tracking down these numbers (including the unavailability of the Gini index & any kind of a recently updated or transparent GPI) would serve as an interesting case study.

Update: if the Obama administration is only four years long, then I'll run the numbers based on Bush's yearly average.

Job Creation
Jan 2001: 132.469M jobs / 285.0M pop (464.8 jobs per thousand ppl)
Jan 2009: 134.580M jobs / 304.1M pop (442.6 jobs per thousand ppl)
+2.111M jobs (263k jobs/year)
-22.2 jobs per 1000 people
BLS

GDP Growth
$10.0125T (Q1 2001)
$14.2003T (Q4 2008)
+41%
bea.gov

Poverty Rate
11.7 (2001)
12.5 (2007)
+ 6.8%
census.gov & cbpp

Child Poverty Rate
16.3% 2001
18.0% 2007
+1.7%
CBPP

S&P 500 performance
1,335.63 January 2001
865.58 January 2009
-35%
Political Calculations

Increase in uninsured
2001: 38.4M
2008: 45.7M
+19%
webmd.com

Net Oil Imports
2001: 10900 barrels/day
2007: 12040 barrels/day
+ 10.4%
EIA

Public Debt
2001: $3.3196T
2008: $5.8029T
+74.8%
CBO

Debt as a pct. of GDP
2000 58%
2008 74.6%
+16.6%
wikipedia

Enrollment in four-year college (pct. per capita)
2001: 15.928M (5.59%)
2008: 18.264M (6.01%)
nces.ed.gov

Population (for college & job creation per capita figures)
July 1 2001: 285.0M
July 1 2008: 304.1M
census.gov