Megan Taylor

front-end dev, volunteacher, news & data junkie, bibliophile, Flyers fan, sci-fi geek and kitteh servant

Visualizing Farmers Markets in NYC

<Stream of Consciousness>

Problem: GrowNYC does not have an at-a-glance visualization of what farmers market are open and where they are.

Solutions: Calendar and map visualizations, to answer both “What’s open now” and “What’s open near me.”

Implementation: I’ve never worked with .ical files before, but I would imagine you could scrape the data off the GrowNYC site and run a script to format it to be added to Google Calendar or whatever. I’m more interested in the map visualization. I’ve done some work with KML and GIS before, so I know it would be possible to scrape the data off the site and build a map with different icons showing year-round, open and closed markets.

Sub-Problem: I know how to scrape a single page of data, but have no idea how to go about scraping a series of static pages.

Solution: Google that!

I would like to scrape the data into a Google Spreadsheet, this would enable a lot of versatility and less maintenance (maybe?). Then set the scraper up to run once a week or so to update the spreadsheet. Spreadsheet could easily be used as the basis for both maps and calendars of various types.

Data needed: Each market’s location, active days of the week and hours, special closed days?, location, special events occurring?.

List of markets

Each market then has a page of its own (which contains the data needed!) like so:

97th St Greenmarket

Probably could all be done with Python. Use Beautiful Soup for scraping? How to get scraping results into Google Spreadsheet then? Google Spreadsheets also has a scraping language, I think. Could try to figure that out.

</Stream of Consciousness>

Yes, this is actually the way I thought this through on the way to work this morning.